Chapter 10

Impact

Definition

What are the higher-level and transformative effects of humanitarian action?

Impact examines the effects of humanitarian action from individual and household levels, through to macro and systemic changes to societies. Beyond immediate effects, impact captures the unintended, varied and collective effects of humanitarian action – positive or negative, in the short, medium or long term.

Key messages

  • Impact means different things to different people. Consider the diverse perspectives of people affected by crisis and the goals of humanitarian actors, ensuring impact captures what truly matters to those affected most by crises.
  • Effectiveness evaluates the achievement of the immediate results of humanitarian action; impact evaluates what these achievements (or non-achievements) mean over time at individual, household, community and societal levels.

10.1: Explanation of definition, and how to use this criterion

Explanation of definition

Impact examines the high-level and transformative effects of humanitarian action. This includes social, economic and environmental consequences that unfold over time and that affect individuals, communities and institutions differently. This aligns with the OECD definition of impact as the ‘transformative effect of an intervention’ or the extent to which it brings ‘holistic and enduring changes’ (OECD, 2019: 64).

In essence, the immediate objectives of humanitarian action are to save lives, alleviate suffering and protect people’s dignity. Use the impact criterion to explore the extent to which humanitarian actors’ efforts reduce the needs, risks and vulnerabilities of people affected by crisis, or the reverse. For example, in humanitarian response to severe flooding, cash assistance or psychosocial support can lower stress, improve food security, reduce negative coping strategies and, potentially, enhance household resilience to withstand future disasters. These are measurable and important outcomes that could lead to sustainable change – impact.

First consider the context, informed by analysis of the needs and priorities of affected people. Second, consider the overall objectives of humanitarian action. What constitutes impact can emerge from discussions with people affected by crisis and other stakeholders, and/or your review of context. Impacts can be:

  • unintended: Pay close attention to unintended impacts, both positive and negative. Focus especially on negative impacts that could be significant. This includes, but is not limited to, environmental impacts (see section section 11.3 Environment and climate crisis) and unintended effects on vulnerable or marginalised groups. Assess any potential to fuel grievances or tensions between groups and other behavioural effects of humanitarian assistance.
  • varied: Consider the extent to which outcomes and impacts have varied between different people, groups and communities. Prioritise the voices of communities affected by crisis in your evaluation (see section 11.1 Putting affected people affected by crisis at the centre).
  • collective: Consider the collective (and sometimes cumulative) impact of multiple humanitarian actors within a context (see also Chapter 8: Inter-connection). It is very difficult to isolate the impact of one actor. For example, explore the synergy between different humanitarian programmes and policies, and whether they contribute to overarching goals to improve the well-being of people affected by crisis or strengthen local institutions (see section 11.2 Locally led humanitarian action).

Note, however, that indirect, varied and collective impacts such as changes in socioeconomic and political processes may take many months or even years to become apparent. Other impacts can be detected and measured in a shorter timeframe at the individual, household and even community level. Determine the timeframe to be evaluated.

When to select impact

Impact is key to understanding if humanitarian action is truly making a meaningful difference, especially from the perspective of those affected by crisis. Use it to uncover indirect positive or negative transformative effects, especially on vulnerable and marginalised people, groups and communities, or on the environment. Evaluate impact to ensure that humanitarian actors adhere to the principle to ‘Do No Harm’ by identifying and mitigating potential harm or the exacerbation of existing vulnerabilities.

Box 6: Impact evaluation

Impact provides a conceptual lens to evaluate high-level and transformative effects of humanitarian action. Impact evaluation attributes observed changes (usually at the individual or community level) to specific humanitarian programmes or projects, using a counterfactual. Thus, it establishes that humanitarian action has directly caused these outcomes.

Impact evaluation and the impact criterion can serve complementary purposes – it is the level of analysis and methods that usually differ. Manage expectations among all stakeholders of what is technically feasible for the depth of outcome and impact analysis. This guides your evaluation approach and cost implications.

How impact relates to other criteria

Use the effectiveness criterion to evaluate what immediate effects have been achieved and for whom. Use the impact criterion to explore the consequences of those achievements – or the lack thereof. In other words, effectiveness tells us what has been accomplished; impact asks so what?

For example, in a cash assistance programme, use effectiveness to assess how populations affected by crisis have used the money – e.g. if families have been able to afford nutritious meals. Use impact to examine if this has led to broader changes – improved nutrition among the targeted population, enhanced well-being or stronger local economies.

Importantly, examining impact also opens up questions about the sustainability of humanitarian outcomes (see Box 3). It prompts us to consider if positive changes – improved well-being or local economic recovery – have been short-lived or have contributed to longer-term benefits for populations affected by crisis. Understanding these dynamics can help identify the types of support that are more likely to lead to lasting change, even beyond the immediate crisis response.

10.2: Shifting the lens: power and positionality

Reflect on how your own identities, assumptions and institutional mandates shape what you consider to be ‘impactful’. Is the presentation of impact primarily shaped by a desire to demonstrate organisational success – potentially at the expense of acknowledging complex or uncomfortable outcomes? This ties to the bias of adopting deficit-based framings when reporting broader or transformative effects. Be alert to when you unintentionally reinforce stereotypes, such as portraying communities affected by crisis primarily as vulnerable, passive or dependent.

10.3: Methodological implications

Methodological Implications - Impact A
Methodological Implications - Impact B

10.4: Evaluation example

Final evaluation of emergency health care services provision for Syrian refugees in Jordan (2021)

Link to evaluation

Background

Islamic Relief Worldwide commissioned an evaluation of its Healthcare Aid for Syrian Refugees in Jordan project. This provided Syrian refugees and vulnerable Jordanians with access to primary, secondary and tertiary healthcare, including haemodialysis services for end-stage renal disease patients. The project also sought to raise health awareness in the community to reduce mortality and improve overall health outcomes.

How the evaluation addresses varied and unintended impacts

The evaluation analyses multiple dimensions of impact: long-term impacts, impacts across sub-groups (varied) and unintended impacts.

-- Using data from surveys and focus group discussions, it assesses if the project created long-lasting and transformational effects for participants. Many patients experienced notable improvements in health following surgical support.

-- Statistical significance tests at the 90% confidence level identify differences in outcomes across sub-groups (e.g. nationality, age, gender).

-- The evaluation shows unintended impacts, particularly increased awareness about COVID-19. Participants were initially hesitant about vaccines or sceptical of the virus, but many felt reassured and more informed having attended healthcare and awareness sessions. Trust in medical staff shifted perceptions and increased vaccine uptake during the pandemic.

Source: Phoenix Center for Economics and Informatics Studies (2022).

10.5: Humanitarian principles and impact

This criterion provides an opportunity to evaluate the wider impact of humanitarian actors adopting (or failing to adopt) a principled approach. Just as it is easier to evaluate the collective impact of multiple humanitarian actors versus single actors, so it is easier to evaluate the impact of principled humanitarian action across the whole response, for example in an inter-agency humanitarian evaluation. However, it is difficult to build sufficient evidence to identify conclusively the wider impact of principled humanitarian action, or of trade-offs made. In-depth research may be more appropriate in some contexts.

At a minimum, explore if and how humanitarian principles have been built into the theory of change. How was principled humanitarian action expected to have an impact? Or has this been overlooked?

Example evaluation question for a joint inter-agency humanitarian evaluation:

To what extent has there been collective effort to follow humanitarian principles, and what has been the overall impact?