Last year, I spoke to United Nations Relief and Works Agency for Palestine Refugees(UNRWA) staff from the Department of Internal Oversight Services about their rapid, real-time operational reviews in crisis settings. Given the intense pressure on staff both affected by crisis and involved in the reviews, we were interested to learn which approaches were most appropriate and useful for creating space to pause and reflect.
Reviews are often intended to support learning, capture insights and challenges, and provide actionable feedback to management. However, the reality on the ground shapes what is possible - it shifts expectations and demands creativity.
Here, I reflect on key takeaways described during an online call with the evaluation team, sharing insights from their most recent review for those considering similar real-time learning exercises in crisis settings.
Connectivity challenges & adaptive communication
In contexts where evaluators cannot access affected populations, connectivity has a major impact. Frequent and widespread blackouts make communication difficult, exacerbated by limited electricity, broken phones and the financial difficulty of topping up prepaid mobile services. In emergency contexts, where much information exchange happens by word of mouth, structured interviews or surveys are severely constrained.
Traditional interview methods are often unfeasible, so evaluators shared that they relied on WhatsApp messages and voice notes for conversations that lasted ten days to two weeks to allow for responses whenever connectivity permitted.
While some staff preferred structured meetings via Microsoft Teams, many found WhatsApp more practical. However, it presents its own challenges: varied response times, off-topic conversations, and blurred personal-professional boundaries for the evaluation team.
Maintaining a coherent narrative requires patience, but the flexibility it affords is invaluable. A dedicated number for these communications, instead of evaluators using their personal phones, may help set clearer limits and allow for a formal closure to the engagement, for example with a final message and details on any follow up.
Light, flexible and impactful methodologies
Instead of being guided by rigid evaluation methodologies, the team chose lighter data collection, focusing on staff perceptions rather than exhaustive triangulation. The core philosophy was simple: if staff overwhelmingly perceived a problem, management needed to address it, even without multiple data sources.
Rather than waiting for a polished report, the team shared findings early through presentations to senior leadership. They also used a communication tree to clarify recipients of findings and offered ‘considerations’ instead of ‘recommendations’ to senior leadership, highlighting positives at the start and end of communications.
Adapting methodologies, from data collection to dissemination, to the demands of the context is a priority. Short formats, delivered through frequently used communication channels and tailored to the reader may facilitate quicker decision-making. For example, following presentations to leadership with a single-page email listing key challenges, successes and priority actions.
Considerations for people interviewed and their stories
In an acute crisis context, the evaluators recognised the emotional and psychological toll on staff, many of whom were displaced, and aimed to avoid adding burden on those that engaged in interviews.
The team resisted sanitising findings for palatability, instead valuing raw, unfiltered perceptions. The review was framed as a learning exercise - not a strict accountability mechanism - creating a space for staff to voice their experiences and reflections.
Acknowledging the complexities of the situation rather than attempting to simplify findings for the sake of easier analysis requires a set up that considers the toll on people interviewed while allowing their stories to emerge.
Real-time learning guided by demand and emerging themes
The team were struck by the demand for structured reflection from staff at all levels - an affirmation of the critical role that real-time learning exercises can play, even in the most challenging settings.
They were guided by staff in evolving their approach; initially structured around standard coordination functions with specific criteria, it shifted to staff care, duty of care and internal communication. Inductive methods emerged with questions that brought out actionable feedback, such as “What needs to change for you to do your job better?”
People value the need to document, reflect on and learn from crisis response. Building on this demand, real-time approaches guided by emerging rather than pre-imposed themes are more likely to result in relevant, actionable findings.
Conclusion: Takeaways for future real-time learning exercises
- Prioritise people over process: Flexibility is key - methodology must adapt to realities on the ground.
- Communication matters: Be creative in choosing channels that work, even if unconventional.
- Protect evaluators and staff: Anticipate the emotional toll and put in place boundaries safeguards.
- Deliver insights rapidly: Focus on timely, actionable feedback rather than polished reports.
- Capture perception as reality: Staff perceptions shape operational effectiveness. Address them alongside other decision-making priorities.
Ultimately, the experience shared reaffirmed the fundamental purpose of real-time learning: to provide space for reflection, to surface critical insights and to drive immediate learning and adaptation. In a crisis setting, the ability to listen, learn and act quickly is more important than rigid adherence to traditional frameworks. Often, the most valuable insights come simply from making the effort to listen.
