When donors and aid agencies concluded their Grand Bargain in 2016, a commitment was made to reduce individual donor(-commissioned) evaluations. Supposedly, the intention was not only to save costs and increase efficiency, but also to move in the direction of joint evaluations. Joint evaluations tend to have a broad scope, looking at groups of agencies or the humanitarian system as a whole. This fits well with the emphasis that has been put on collective performance since the reform efforts of the 2012 Transformative Agenda and 2016 Grand Bargain.
The real-time review of the international response to hurricane Matthew in Haiti – the first large-scale crisis following the 2016 World Humanitarian Summit – was a first, promising step towards living up to the Grand Bargain commitment to reduce duplication in evaluations. Carried out under the auspices of a group of donors and agencies and using the Grand Bargain as analytical framework, this evaluation looked at the system as a whole, including donor performance. Sadly, this initiative has not been repeated. Two years later there lacks a systematic and consistent approach to joint humanitarian evaluations. They are few and far between at a time when there is no shortage of large-scale, so-called L3, crises.
Looking at the Rohingya refugee crisis, the system is nearly back at square one when it comes to joint evaluations. Very few of the increasing number of evaluations done in this context review joint agency performance and look deeply at broader collective challenges that hamper a more effective overall response. Joint evaluations would be highly appropriate however given the extreme complexity of the issues at hand. Inter-agency coordination problems, life-threatening congestion in the camps, and the lack of a refugee protection framework cannot be addressed by any single organisation.
That said, the major UN agencies that chose to carry out evaluations have done so separately, each with different approaches, methodologies, and objectives.
Photo credit: Dutch Ministry of Foreign Affairs
The traditional axiom applies among them: let’s get our own house in order first. The competition between UN agencies for a lead role in this response certainly did not help to create an atmosphere of collectiveness. If their evaluations will be shared, this will likely only happen once the reports have been put in the public domain. This is long after the moment when they could have shared first findings, developed joint analysis, and formulated collective answers.
In terms of evaluating NGO responses for the Rohingya refugees, the picture is slightly better. As a joint fund-raising mechanism for UK-based NGOs, the Disasters Emergencies Committee (DEC) is rightly concerned with the collective work of its members. The DEC real-time evaluation (RTE) carried out in the first quarter of 2018 drew attention to the general context of the refugee crisis.
Photo credit: Flickr/DFID
It found that without addressing the three dominant contextual factors the responses of DEC members (and other organisations) would risk losing their relevance. Other European NGO networks launched separate initiatives recently to have their members’ collective work evaluated. As evaluators are likely to interview a number of the same representatives from Government, UN agencies, and NGOs (not to mention refugees) and develop comparable analyses, they may consider consolidating their evaluation exercises.
Joint exercises are moreover indispensable for understanding whether there has been progress towards ‘working as a system.’
This is not to say that individual agency reviews do not serve a purpose. On the contrary, agencies do need to assess the quality and effectiveness of their operations, but for such an exercise to be meaningful and relevant, it must take the bigger picture into account. The plight of Rohingya refugee women and girls does not end at the doorstep of a therapeutic feeding centre, especially when they run the risk of rape and/or being taken to the hotels that serve as brothels in Cox’s Bazar. If joint evaluations are not taking place in this crisis, where interdependence among humanitarian actors is so intensely felt, this does not bode well for other crisis responses.
Joint evaluations reviewing collective performance are not an easy undertaking. Too often, far too much time is lost in negotiating terms of reference, accepting findings, or developing recommendations. These issues can be solved by narrowing the focus to top-line issues. Joint exercises are moreover indispensable for understanding whether there has been progress towards ‘working as a system.’ For the humanitarian reform efforts such as the Transformative Agenda and the Grand Bargain not to be wasted, collectively, donors and agencies should give more political weight to assessing joint results on the ground.
