Rethinking the Impact of Humanitarian Aid 24 th

Re-thinking the Impact of Humanitarian Aid 24 th Biannual Meeting Berlin, 2 nd December 2008

ALNAP 24 th Biannual Aims n To help clarify key issues around use of humanitarian impact assessments n To move towards a shared understanding of the limits and possibilities of humanitarian impact assessment n To use this understanding to outline a practical vision for future work in this area 24 th ALNAP Biannual Meeting, December 2008 2

Overview n Humanitarianism ‘transformed’ and the emergence of impact assessment n Challenges for assessing impact n Conclusions 24 th ALNAP Biannual Meeting, December 2008 3

Humanitarianism Transformed Three broad trends since the 1980 s contribute to the current interest in and debate around humanitarian impact assessment: ¨Humanitarian aid expanded and politicised ¨Humanitarian aid institutionalised and professionalised ¨Changing nature of vulnerability and human suffering 24 th ALNAP Biannual Meeting, December 2008 4

Increase in initiatives, across and within agencies n Cross agency efforts include SMART; HNTS; TRIAMS; Fritz Humanitarian Impact Project; the CDA Listening Project; ALNAP HPP; SPHERE n Quality Compass; ECB ‘Good Enough’ Guide to impact measurement; DEC’s new Accountability Framework etc n WFP, ECHO and UNICEF all include impact in their evaluation guidelines; Action. Aid’s ALPS; Save the Children UK’s GIM 24 th ALNAP Biannual Meeting, December 2008 5

But despite considerable progress, problems remain. The Biannual background paper has identified six key broad challenges: 1. Defining impact assessment 2. Diverse stakeholders and interests 3. Indicators, baselines and data 4. Methodologies 5. Collective interpretation and analysis 6. Capacities and incentives 24 th ALNAP Biannual Meeting, December 2008 6

1. Defining ‘impact’ and impact assessment 24 th ALNAP Biannual Meeting, December 2008 7

Ideal picture of impact: the MDG Goal of universal primary education by 2015 24 th ALNAP Biannual Meeting, December 2008 8

Reality of impact is rather different… “…Universal primary education would be achieved at present rates of progress in 2079 in sub-Saharan Africa and in 2036 in the Middle East and North Africa…” Social Watch 2007 24 th ALNAP Biannual Meeting, December 2008 9

Real world impact is complex Activity Output Outcome Impact 24 th ALNAP Biannual Meeting, December 2008 10

and hard to discern, even a long time after the fact 24 th ALNAP Biannual Meeting, December 2008 11

Attribution or Contribution? Private Sector Other NGOs A. N. NGO Local partners Developing Country Govmts 24 th ALNAP Biannual Meeting, December 2008 Community and Family Religious organisations Civil Society 12

A widely recognised definition of impact assessment “…Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions…. ” Novib/Oxfam research project, reported in C. Roche, Impact Assessment for Development Agencies 24 th ALNAP Biannual Meeting, December 2008 13

Humanitarian action has its own challenges n Lack of clarity on IA definition and purpose n Rapidly changing humanitarian contexts n No consensus on objectives of humanitarian aid n Intended impacts of interventions often unclear or overambitious 24 th ALNAP Biannual Meeting, December 2008 14

Contextual differences between normal development aid and humanitarian aid situations Development n Considerable Lead Time n Deliberate & pro-active n Will take time, be thorough, extensive with comprehensive data collection n Location chosen n Duration planned n Beneficiary population identifiable and static n IA goals may be made compatible with socioeconomic ones 24 th ALNAP Biannual Meeting, December 2008 Humanitarian n Sudden onset n Reactive n May need to be partial in coverage n Unpredictable location n Uncertain duration n Beneficiary population heterogeneous and dynamic n Priority given to “life saving” activities sometimes difficult to reconcile with IA goals

2. Diverse stakeholders, interests and objectives 24 th ALNAP Biannual Meeting, December 2008 16

24 th ALNAP Biannual Meeting, December 2008 17

The challenge of stakeholders (illustrative) UN agencies Donors International NGOs National and local partners Media Affected Population Red Cross / Red Crescent 24 th ALNAP Biannual Meeting, December 2008 Political authorities Private Sector Military 18

Different stakeholders have different perceptions of and interests in impact n Enabling different stakeholders to express divergent views of impact is crucial to successful impact assessment n IA findings more likely to be used they meet the interests of the end users 24 th ALNAP Biannual Meeting, December 2008 19

Different needs may not be reconcilable and achievable in a single impact assessment: Accountability or Learning? n “The purpose of most impact assessments is to demonstrate past impact and to improve future practice, and there may be tension between the two. ” n Often, too much is expected: “If we continue to expect evaluation to cover most of the accountability needs of the sector, we will be disappointed” (Sandison, P. 2006). 24 th ALNAP Biannual Meeting, December 2008 20

3. Indicators, baselines and data 24 th ALNAP Biannual Meeting, December 2008 21

Indicators, Baselines and Data for humanitarian IA Identifying impact indicators involves value judgements about what kinds of changes are significant for whom (Roche, C. 2000) 24 th ALNAP Biannual Meeting, December 2008 22

‘The familiar adage [“you can lead a horse to water, but you can’t make it drink”] illuminates the challenge of committing to outcomes. The desired outcome is that the horse drinks the water. Longer-term outcomes are that the horse stays healthy and works effectively. But because program staff know they can’t make a horse drink water, they focus on the things they can control: leading the horse to water, making sure the tank is full, monitoring the quality of the water, and keeping the horse within drinking distance of the water. In short, they focus on the processes of water delivery rather than the outcome of water drunk. Patton, M. 1997: 157 -8 24 th ALNAP Biannual Meeting, December 2008 23

“Reports were so consistent in their criticism of agency monitoring and evaluation practices that a standard sentence could almost be inserted into all reports along the lines of: It was not possible to assess the impact of this intervention because of the lack of adequate indicators, clear objectives, baseline data and monitoring. ” (ALNAP, 2003) 24 th ALNAP Biannual Meeting, December 2008 24

Issues include… n Weak or non-existent baselines n Data is often unavailable or unreliable n Data collected is mainly quantitative n Monitoring systems focus on process and outputs 24 th ALNAP Biannual Meeting, December 2008 25

4. Methodologies 24 th ALNAP Biannual Meeting, December 2008 26

Wealth of tools, techniques and approaches are available n Documentary analysis n Interviews n Questionnaires (including recipient perceptions surveys) n Monitoring n Ex-post evaluation n Case studies n Participatory Rapid Appraisal (PRA) n Experimental / quasi-experimental 24 th ALNAP Biannual Meeting, December 2008 27

Qualitative versus quantitative n Any research design is shaped by both opportunities and constraints n Quantitative methods able to tackle ‘what’ and ‘where’ questions n Qualitative methods able to answer ‘why’ and ‘how’ questions, and are good at capturing process 24 th ALNAP Biannual Meeting, December 2008 28

24 th ALNAP Biannual Meeting, December 2008 29

Mixed methods approaches can take into account, rather than dismiss, the complexity of assessing humanitarian impact 24 th ALNAP Biannual Meeting, December 2008 30

5. Collective interpretation and analysis 24 th ALNAP Biannual Meeting, December 2008 31

Improved interpretation and analysis of data through engagement with affected populations and other stakeholders n Humanitarian impact should not only be about providing more and better information, but also about making sure that findings are used in ways that improve the lives of affected populations n Wider stakeholders should also be engaged in this process ¨ “Learning partnerships” for impact assessment 24 th ALNAP Biannual Meeting, December 2008 32

To date, participation by and accountability to affected populations has not been a key feature of impact assessments Attempts to improve this include: ¨ ECB ‘Good Enough Guide’ ¨ The Quality Compass ¨ Feinstein International Center Participatory Impact Assessments (PIA) 24 th ALNAP Biannual Meeting, December 2008 33

6. Capacities and incentives 24 th ALNAP Biannual Meeting, December 2008 34

Capacities and Incentives for improved humanitarian impact assessment n Lack of individual and organisational capacity to do good impact assessments ¨ TORs are often unclear ¨ objectives are not defined clearly within the context of the intervention ¨ stakeholder analysis is limited ¨ timing relates to institutional priorities rather than humanitarian need ¨ skills relating to impact assessment methodologies are lacking n Contributing factors: high staff turnover; lack of a learning culture; inadequate investment and resources 24 th ALNAP Biannual Meeting, December 2008 35

There is a considerable lack of incentives n Institutional incentives can override humanitarian ones; too few incentives to conduct good impact assessments; results-based approaches can create perverse incentives n A number of cultural barriers and biases that hinder good quality humanitarian impact assessment 24 th ALNAP Biannual Meeting, December 2008 36

Recap: six challenges 1. 2. 3. 4. 5. 6. Defining impact assessment Diverse stakeholders and interests Indicators, baselines and data Methodologies Collective interpretation and analysis Capacities and incentives 24 th ALNAP Biannual Meeting, December 2008 37

Conclusions “…Taken as a whole, the humanitarian system has been poor at measuring or analysing impact, and the introduction of results-based management systems in headquarters has yet to feed through into improved analysis of impact in the field… it is arguable that there has been significant under-investment in evaluation and impact analysis…” (Hoffman C. A. et al, 2004) n Our review gives little indication that there has been much movement from the position above, articulated in 2004 n Can we move forward? How? 24 th ALNAP Biannual Meeting, December 2008 38