Introduction

This guide updates the ALNAP guide Evaluating humanitarian action using the OECD DAC criteria, first published in 2006 to help evaluation professionals better understand and apply evaluation criteria in humanitarian settings (ALNAP, 2006). While this edition retains the same seven evaluation criteria, we have updated, expanded and clarified the definitions to reflect changes in evaluation practice, shifts in the humanitarian system and feedback from evaluation practitioners. The updates also incorporate revisions made by the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD DAC) in 2019 (OECD, 2019, 2021). See Box 1 for a brief timeline of the OECD criteria and how they have been adapted over time for the evaluation of humanitarian action (EHA).

As the humanitarian system and the world around it has changed, the need for high-quality evaluative material remains vital to the humanitarian community. To meet this need, now and in the future, it has been critical for us to understand the diverse ways that the criteria are interpreted and applied in EHA. Therefore, the revision has been informed by a participatory and evidence-based process.

Consultation events were held in different languages and multiple locations, and an online survey was available in English, Arabic, Spanish and French. Several events were held with voluntary organisations for professional evaluation (VOPEs), including the Asia Pacific Evaluation Association (APEA), the Lebanese Evaluation Association (LebEval) and the Réseau Francophone de l’Évaluation (RFE). ALNAP reviewed how the criteria are used in EHA and some of the challenges in their application (ALNAP, 2023). An advisory group supported the entire process, and a reference group fed back on the draft guide. Members of both groups represented a wide array of organisations, including donors, UN agencies and non-governmental organisations (NGOs) based in the Global North and in Global Majority countries. Further details on the revision process are available on ALNAP’s website.

The consultation process confirmed the enduring relevance of the OECD evaluation criteria plus clear demand for definitions and guidance tailored to EHA. Stakeholders recognise that the criteria provide a valuable common language across diverse actors and contexts, but the criteria require thoughtful interpretation to remain useful, particularly in crises.

The revision process also drew attention to wider social and political shifts that are shaping the humanitarian system. These shifts lay bare the need for humanitarian action to respond to global challenges and deliver on long-standing reform commitments. As the number of complex and protracted crises grows, and the funding environment changes, we must re-examine the structures and assumptions underpinning humanitarian action – particularly those shaped by institutions in the Global North. Greater attention to racism, colonial legacies and unequal power relations in the humanitarian system also raises important questions for EHA (ALNAP, 2022). Whose knowledge counts? Who decides what gets evaluated and how? And who defines success?

Our stakeholders want EHA to give more explicit attention to these evolving issues. While evaluation is intended as a learning and accountability tool to improve performance, the utility of EHA can be enhanced by addressing areas of humanitarian action where profound changes are needed at a system level.

Accordingly, this guide introduces three priority themes – putting affected people at the centre, locally led humanitarian action, and environment and climate crisis. This is not an exhaustive list, but it is indicative of where clearer and more consistent focus in evaluations could support transformational change of the humanitarian system, in addition to fostering incremental change (Quinn Patton, 2020). Include these in evaluations as standalone themes or embed them under the criteria – they are intended to inspire you to follow meaningful lines of enquiry. See Chapter 2 for more on the link between evaluation criteria and priority themes, and see Chapter 11 on each theme.

This guide provides practical and accessible support on how to use and interpret the OECD evaluation criteria in EHA. It complements the OECD guidance on using evaluation criteria, in recognition of the multitude of ways of ‘doing’ evaluations. Primarily, it is intended for evaluators of humanitarian action and those involved in commissioning, designing and managing EHA. It is also a key resource for training courses on EHA. And it is a reference for humanitarian evaluation users, from practitioners to policy-makers.

Read the guide from start to finish for an overview of the criteria and how best to apply them in an evaluation. Or use it to understand the scope of individual criteria and whether they are applicable to the evaluation in hand. Alongside this long-form guide, a summary with key take-aways is available on ALNAP’s website.

The guide describes methodological considerations for each criterion. However, this is not a methods guide, and it does not detail wider issues such as evaluation management, types of evaluation approaches, and methodologies and tools for data collection. To ensure longevity, the guide mostly uses common language instead of terminology that may become outdated, e.g. accountability to affected people. Please see ALNAP’s EHA guide for comprehensive guidance on EHA (ALNAP, 2016) and the companion guide on the Evaluation of protection in humanitarian action (ALNAP, 2018).