This section describes the overall research approach for the SOHS Report, followed by an overview of the methods used for data collection and analysis. A more detailed description of the draft framework guiding the performance assessment is provided in Section 3.3, and a detailed description of individual research components is given in Section 4.
3.1 Research approach
The SOHS reports together create a longitudinal study using a relatively consistent mixed-methods research approach over time. The sixth edition will make some shifts - in both the research questions that structure the framework of the report and some of the research components used to provide the data – to support relevance and accessibility to practitioners and policy makers and to reflect a greater diversity of voices and perspectives.
3.1.1 Adaptive approach to the sixth edition
To inform the inception phase and study design, we engaged a range of stakeholders in a set of four multi-actor consultations in April 2024, including several representatives from local NGOs and networks. However, further efforts will be made to strengthen the voices of crisis affected people and local humanitarian actors in the design of the report. Continuing the approach of the fifth edition a small number of initial focus group discussions will be conducted as part of the in-country research (described in Section 4) to help refine the study matrix and ensure the report asks questions that crisis affected people consider most important in this period. The co-leads will also engage with the local in-country researchers during the initial stages of research to understand key themes, trends and priorities from their perspectives – those discussions will feed both into the study matrix for the global report and also inform the direction of a set of country-specific reports (also discussed in Section 3.1.2).
This adaptive approach means the study matrix for the report will evolve in the months following the publication of this inception report. A full study matrix that reflects this evolution – including indicators – will be published online a quarter of the way through the primary data-collection phase. This Inception Report provides the initial themes and research questions, which will be adapted based on initial exploratory research. The final analysis and writing of the report will allow scope for shifting the direction of the specific chapters and the overall framing of the report to best reflect the evolving understanding of the system over the study period and emerging trends, crises, or influential policy events. Periodic reviews will be held to adjust the research approach accordingly.
3.1.2 Engaging more diverse perspectives to inform the report and its impact
The sixth edition will make additional efforts to engage a diversity of perspectives within the research, analysis and writing processes through four main shifts.
While maintaining existing members of the SOHS Support and Advisory Group (SAG) that have been integral to previous reports, we have extended invitations to more representatives of organisations from global majority contexts to join the SAG to provide a greater diversity of perspectives at key stages of the research cycle, from inception through to dissemination. In addition to increasing the LNA membership of the SAG, we have invited development and peace actors to the group to ensure advice is received from key stakeholders engaging with humanitarians across the HDP nexus.
We have also sought stronger partnerships with local researchers in the qualitative in-country research component, asking them to engage more actively in the design and analysis that contributes to the global research report but also in developing tailored country reports. We aim to approach these relationships with a view to more equitable partnerships than in the past, providing space for greater influence of local researchers in determining the key issues to explore in their context. The production of local reports should also lead to a less extractive approach than focusing only on the production of the global report, providing a tailored resource that can be used for local influence. A new position has been hired to focus on that component and the equitable partnership approach: a Senior Research and Partnership Coordinator.
The sixth edition will continue the efforts made in the last report to represent the perspectives of crisis affected people throughout the research cycle. The design of the research will be informed by initial FGDs and by an analysis of existing data that captures their perspectives. For example, we will analyse recent qualitative and quantitative data gathered from people affected by crisis by Ground Truth Solutions. Beyond the research design, these voices will continue to inform the research findings through two regular SOHS research components: in-country qualitative data collection and the aid recipient survey (described in Section 4).
Finally, we have sought to diversify the research and writing leadership within the core SOHS team. In recent years, leadership and coordination of the report has been provided by global minority researchers. ALNAP’s new approach to global recruitment has allowed us to hire a Senior Research and Partnership Coordinator based in Kampala, Uganda, to support the SOHS coordination and development of country reports, and the lead authorship team will be expanded to reflect greater diversity of perspectives.
3.1.3 Assessment of performance and progress over time
The SOHS Report is a longitudinal study which both assesses performance for a particular period of time and also compares this performance against previous periods to understand whether – on the whole – humanitarian policy and action is improving, declining or remains unchanged. Earlier editions had largely taken a deductive approach, whereby the qualitative and quantitative data is collected, analysed, and triangulated over a period of 18 months following a pre-existing set of performance criteria drawn from the OECD DAC evaluation criteria that reflect the different quality measures humanitarian action is expected to meet. The first four editions of the report structured their research questions around these set performance criteria; however, the fifth edition altered that approach by framing the research questions in the form of policy-relevant questions that were viewed as more accessible to a non-evaluative audience. For example, the research question and chapter heading “Does humanitarian action work?” largely maps on to the DAC criteria ‘effectiveness’. Despite the shift in the chapter framings, the longitudinal comparison of the DAC criteria over time was retained and presented in a table in the conclusion. The sixth edition will maintain the approach of the last report by largely using policy relevant research questions to create an accessible structure and retaining the production of the longitudinal DAC criteria table in the conclusion to support continuity. The policy questions will speak to issues of contemporary importance and will link to the important challenges and demands presented by the external operating environment in this study period, 2022-25. The descriptive discussion of key trends, crises, and the financial size of the system in the opening section of the report will help to situate the performance of the system within that operating environment.
Table 1 outlines the draft set of policy research questions and how the DAC criteria can be loosely mapped within their structure to enable the creation of the longitudinal table in the conclusion. Qualitative and quantitative data will be collected, analysed, and triangulated concurrently over a period of 18 months according to a set of more detailed sub-research questions (see Tables 1, 2, and 3) and a list of qualitative and quantitative indicators, which will be provided in the detailed SOHS study matrix. The SOHS report itself will not necessarily be structured according to this framework but will be designed to present the data and findings in an accessible and compelling way to inform policy and practice.
3.2 Overview of data collection and research components
This section describes the eight consistent research components, or methods, used in the SOHS research approach and the newer addition of thematic studies that were introduced in the fifth edition to fill key evidence gaps across the different policy questions that structure the report.
3.2.1 The core research components
Since 2012, each edition of the SOHS Report has drawn on eight methods of data collection and analysis. Data collection across these eight components is integrated, using a shared research framework outlining the questions and indicators related to each area of the performance assessment. The draft research framework for the sixth edition is described in Section 3.3, and the individual research components or methods are described in further detail in Section 4.
Primary data collection and analysis
In-country research: Focus group discussions (FGDs) and key informant interviews (KIIs), along with relevant context-specific documentation and observations, are collected for a minimum of five crisis contexts over the study period.
Aid recipient survey: ALNAP conducts a survey of aid recipients in four to six crisis contexts to elicit their assessment of humanitarian performance. In the past, these have been conducted using SMS text message, interactive voice response (IVR) and computer-assisted telephone interviews (CATI)-based modalities. The previous edition surveyed 5,487 aid recipients.
Practitioner survey: An online survey is used to elicit the perceptions of humanitarian practitioners on humanitarian performance. Previous editions also sought to capture the perspectives of host-government representatives through a similar survey approach. Recent attempts to capture government perspectives via the survey had very limited response rates. As such, the 6th edition will only conduct the survey with humanitarian aid practitioners and will access the perspectives of host government representatives via key informant interviews at the in-country and global levels.
Key informant interviews (KIIs) at global level: Humanitarian leaders and key thinkers from multiple contexts are interviewed to assess performance and identify important trends. These interviews are also used to identify potential data sources to address key evidence gaps and to triangulate and test hypotheses emerging from other sources of data.
Organisational mapping and analysis: Data is collected from individual organisations as well as through a desk-based review to provide an overall picture of the number of humanitarian staff and organisations worldwide, their makeup, and their expenditure.
Analysis and synthesis of secondary data
Evaluation synthesis: A synthesis of findings from humanitarian evaluations published in the study period is conducted, with the aggregate findings weighted according to quality.
Financial analysis: ALNAP works with experts in humanitarian financing to produce and analyse statistics on humanitarian finance flows and compare this to previous SOHS Report periods.
Literature review: A review is conducted of research reports and academic work published within the study period on a sub-set of themes related to humanitarian policy and practice.
3.2.2. Thematic studies
The fifth edition of the SOHS Report adopted a new methodology in the commissioning of a set of primary research studies to fill key evidence gaps to help assess the performance of the system where existing data was lacking. These were also designed to provide more empirical data to complement the perception-based assessment of performance provided by other existing research components.
The thematic studies commissioned for the fifth edition provided useful insights that helped to strengthen the analysis across several performance components – including around effectiveness, efficiency, complementarity, and connectedness – and to strengthen the original knowledge contribution of the SOHS research. The sixth edition will continue with commissioning some select original research. It will, however, alter the approach by commissioning a reduced number of original thematic studies as independent research. The last report commissioned five individual primary studies,[4] which spread the available resources across different themes in a way that limited the depth of data gathering and analysis that could be achieved for some of the particularly challenging areas. For example, it was difficult to adequately assess the contribution of humanitarian action to reduced mortality with available time and resources. While findings from each of the studies provided some data that could be incorporated into the SOHS report, there was not enough depth and rigour to answer some of the questions we would have liked to answer and only the innovation study provided the depth of information to support publication of a standalone report.
The sixth edition will commission only 1-2 primary thematic studies to allow for more substantial depth in the data gathering and analysis. The publication of any reports that come from those studies will be staggered around the launch of the main SOHS synthesis report. In some cases, a thematic issue may not be a priority for a new independent thematic study but may still require additional investment to answer key questions in the main SOHS report. For these thematic issues, some additional data gathering will be incorporated into existing research components. For example, via additional investment in new financial statistics or via additional targeted KIIs and literature reviews.
Given the more targeted investment in a smaller number of commissioned thematic topics during a constrained funding environment for humanitarian research, careful prioritisation of where ALNAP can add most value to the sector via this new primary research is essential. ALNAP held a series of four multi-sector consultation events in April 2024 to identify topics of greatest contemporary interest to the sector, where original research would add value for policy or practice, and that would avoid overlap of research planned by other organisations over the study period. At these consultations we presented eight thematic concept notes [5] for discussion and for collective prioritisation. The direction chosen for these studies is described in section 4.9.
3.3 Research framework
The role of the research framework is to guide data collection and analysis. When complete, the full research framework comprises five components: (i) research questions; (ii) study matrix; (iii) coding matrix; (iv) data-collection tools (e.g. interview protocol); and (v) interpretation and analysis plan.
The following section outlines the research questions (i). The study matrix (ii) will be made available online in early 2025 and a final version will be published alongside the final report.
3.3.1 Assessing performance: key policy-relevant performance questions and the DAC criteria
The sixth edition of the SOHS will build on the approach of the fifth edition by constructing the research framework around key policy-relevant performance questions. As noted above, there is scope for these questions to be refined during the data collection and analysis phase in line with shifts in trends, crises, and policy events. They can be considered placeholders in this inception report outlining the proposed approach.
This framework continues to allow longitudinal assessment of the DAC criteria because those criteria are nested within the different policy questions of the research framework (see Table 1). The SOHS was originally designed to provide a longitudinal assessment and accountability tool for the sector, with the DAC criteria providing a useful structure to allow that tracking over time. After two decades, the DAC criteria remain a key common structure for evaluating humanitarian action, however, conceptions of what is means to engage in good quality humanitarian action have shifted over time and there have been successive alterations by the OECD DAC on the criteria. The SOHS has sought to reflect some of those shifts by slightly adapting the DAC criteria in successive reports.
The OECD DAC published a revised set of evaluation criteria for the development community in 2020 and ALNAP has engaged in a series of multi-stakeholder consultations to update the definitions and guidance on applying these criteria in humanitarian evaluations. These updated definitions and guidance will be published in 2025 and will be reflected in this sixth edition of the SOHS. We expect to largely maintain the ability to support longitudinal comparison using these updated criteria because the adjustments to the adapted criteria used in recent SOHS reports are expected to be relatively small. As the guidance is finalised, we have presented the criteria used in the last report in the table below and will update the phrasing when the new guidance is finalised.
3.3.2 Assessing performance at programme and system level
The sixth edition of the SOHS will explore performance of the system at both the programme and system level. It will continue to synthesise evidence on the programmatic performance of the system to provide an aggregate picture of progress at implementation-level for different functions of the humanitarian system. It will also retain the growing focus of the last report on system-level performance, which included an examination of progress against key commitments made in the World Humanitarian Summit and Grand Bargain. It will, however, need to explore the increased number of system change initiatives that have gained prominence over the study period, including Grand Bargain 3.0, OCHA’s Flagship Initiative, the Charter for Change, the Pledge for Change, reforms to the humanitarian program cycle, and CHH-Lancet Commission on Health, Conflict and Displacement, among others. This system level view will be of particular importance given the structural nature of several of the contemporary operational challenges and trends identified in recent consultations. The draft research framework below integrates key questions reflecting both levels of analysis within each main policy relevant research question.
3.3.3 Research framework
The draft research framework will form an initial structure for gathering data and synthesising the findings on system performance. The structure of the public facing report is likely to have different chapter headings and groupings than outlined in this initial framework as our findings help to determine the key messages and how they should be presented to maximise use and relevance for policy makers and practitioners. The framework will be complemented by a more detailed study matrix which will contain the research sub-questions and indicators and match the research components to these questions. Within each area, the report will gather data both on challenges to performance and on where good practice examples have emerged. Each research question will involve a disaggregation of performance for different profiles of recipients, including by sex and age. The experiences of different identity groups – such as people living with disability, migrants or LGBTQI+ individuals – will also be considered within the research, however, presenting a clear disaggregation by other groups may be challenged due to varying levels of data availability.
[4] These studies were focused on the following topics: Innovation, mortality, accountability modalities, localization and the HDP nexus.
[5] Making sense of the development, peace and diplomacy gap; Looking outside the multilateral humanitarian system; Displacement; Humanitarianism in the new age of geopolitics; Protection; Climate Crisis; Efficiency; Sufficiency and the humanitarian prioritisation challenge.