Humanitarian Principles in Evaluation how can we move
- Slides: 9
Humanitarian Principles in Evaluation: how can we move forward? Margie Buchanan-Smith on behalf of Tony Beck (team leader), Belen Diaz and Lara Ressler Horst UNEG EPE 25 April 2016
How did we go about this review? Our approach and methodology • Aim: to provide the HEIG with a better understanding on how the 4 core HPs are evaluated, highlighting best practices, challenges and opportunities • Our methodology: • Literature review • Analysis of humanitarian strategies, evaluation policies & guidelines of 10 agencies • Screening of 142 evaluations of HA for coverage of HPs, focusing on 7 emergencies: Afghanistan, DRC, Haiti, Somalia, Sudan, South Sudan & Syria • Analysis of a purposive sub-sample of 20 evaluations which made greater reference to HPs • Interviews with 12 key stakeholders for reflections on HPs in EHA
What did we find? The evidence • Commitment to HPs in agencies’ humanitarian policies not well-reflected in evaluation policies and guidelines • Review of evaluations: • Widespread discussion of ‘access’ & ‘security’, but the link to HPs tenuous & implicit • Explicit mention of HPs in about one-third of evaluations, but often lacking in-depth analysis • ‘Impartiality’ – the most frequently referenced principle, usually addressed under evaluation criterion of coverage • Comprehensive evaluation of HPs combined not taking place: most frequent combinations: Independence and Neutrality; Independence and Impartiality • Concentration of terms in 20 out of 142 evaluations
Usage of terms in sample of 142 evaluation reports
What did we learn from this review of evaluations? • No significant difference between different types of agencies in their treatment of HPs in EHA • Strategic & thematic evaluations more likely to reference HPs than operational, RTE or impact evaluations • Extremely limited ‘good practice’ of evaluating HPs: 6 out of 142 evaluations • Overall: HPs are not systematically assessed in EHA
What emerged as the challenges to evaluating humanitarian principles? • Lack of common understanding of HPs • How to evaluate when there are contradictory sets of principles • Sensitivity to evaluation of HPs in the public domain • Discussions about HPs take place ‘behind closed doors’ • Methodological challenges: requires evaluation through a more ‘political’ lens. EHA tends to be more ‘technical’ • Lack of expertise to evaluate against HPs • Lack of guidance
What did we learn from the good practice examples? • TOR: have specific questions related to HPs • Political context analysis: in relation to HPs • Methodology: mostly using standard methodologies, with one innovative example • Inclusion in evaluation analysis: 2 ECHO thematic evaluations included a comprehensive analysis of implementation of HPs, bringing out tensions between principles • HPs reflected in recommendations: often quite general, some more specific eg related to access and coverage • Just one evaluation (ECHO, 2012) explores whether adhering to HPs leads to more successful interventions
What else did we learn/ deduce about facilitating factors? • Requires clarity of commitment by the agency to HPs • Requires expertise • Evaluation managers with a good understanding of HPs • Requires expertise on HPs in the evaluation team • More likely in strategic evaluations (and research studies…) • Could be supported with guidance on how to evaluate against HPs • Requires a different approach to EHA: less mechanical, with analysis of political context linked to analysis of HPs • Agencies must be willing to evaluate against HPs
Suggested follow-up for the HEIG 1) Update IAHE guidance on system-wide emergencies with greater attention to evaluation of HPs 2) Do follow-up review of NGO evaluations and how they reflect HPs 3) Pilot evaluation of HPs (where a lesser degree of political conflict) drawing on available methodologies 4) Pilot a confidential HP Annex, supported by peer review 5) Commission single-agency evaluations focussing specifically on HPs 6) Use existing Communities of Practice to disseminate these findings and move the discussion and practice forward 7) Carry out meta-evaluations to see if evaluation practice on HPs improves