Methodological Lessons Joint Evaluation of Conflict Prevention and

Methodological Lessons Joint Evaluation of Conflict Prevention and Peace-Building in DRC, 2002 -2010 Day 2, Presentation in Oslo ‘What Have We learnt From the Application Phase? ’ 17 February 2011

The General Feedback On the Draft Working Guidance 1. Feedback received from the Congo evaluation team, but also from recent work with USAID-OTI, Ausaid, and training given by Channel on evaluation of CPPB. 2. The Guidance is useful introduction, especially for practitioners with limited experience of evaluation. Particularly useful are the overview of key steps and the development of the evaluation questions. 3. Intrigued by Theories of Change and importance of conflict analysis. Find Theories of Change better for project level. 4. However the Guidance does not tackle the “how to” questions: conflict analysis (which, when, how), testing theories of change, baselines. Channel Research 2

The Evaluative Framework Used for the Congo Evaluation Mandate & Steering 1. Evaluation Questions & Criteria Effects on the Conflict 2. Sample of Projects Outcomes 3. Conflict Analysis 4. Research Hypotheses and Themes

The Evaluative Framework Used for the Congo Evaluation Mandate & Steering 1. Evaluation Questions & Criteria Good to have simple questions but tricky to analyse by “results”. Effects on the Conflict 2. Sample of Projects Outcomes 3. Conflict Analysis 4. Research Hypotheses and Themes

The Evaluative Framework Used for the Congo Evaluation Mandate & Steering Issue N° 1: No Theories of Change Confused Strategies 1. Evaluation Questions & Criteria Effects on the Conflict 2. Sample of Projects Outcomes 3. Conflict Analysis 4. Research Hypotheses and Themes

Reasons for the Weakening of Theories of Change & Strategies as Reference 1. Seizing Opportunities 2. Compartmentalisation • Implementation requires a response to constraints and opportunities as they arise. • These cannot be planned for in advance so plans and objectives are not a good evaluation point of reference • Too many levels and too many instruments to carry out an analysis • Severe lack of documentation, issues of disclosure, many “indirect” effects. 4. End State? 3. Problems of Reconstruction • Theories of Change conceived as an elicitive tool in planning • Retrospective interpretation opens up real chances of contested findings, with no clear data sets • Problems in defining terms such as peace and conflict. • Usually depends on the identification of an end-state. • This may not be explicit or shared by all stakeholders or those within a single programme Channel Research 6

The Evaluative Framework Used for the Congo Evaluation Mandate & Steering 1. Evaluation Questions & Criteria Effects on the Conflict 2. Sample of Projects Issue N° 2: No Conflict Analyses Used, and No Conflict Sensitivity Analysis Outcomes 3. Conflict Analysis 4. Research Hypotheses and Themes

Our understanding of peace depends on where we are

The Evaluative Framework Used for the Congo Evaluation Mandate & Steering 1. Evaluation Questions & Criteria Issue N° 3: Finding the right balance in the sample is not easy 2. Sample of Projects Effects on the Conflict Outcomes 3. Conflict Analysis 4. Research Hypotheses and Themes

The Question of Sampling An excessive number of projects. A database selected before the evaluation and a great difficulty in accessing information for these projects. Consequence: Time spent to obtain information: 80% (to avoid focus on + transparent) Sample may not represent the full range of interventions Strategic dimensions were lost. Case study approach weakened.

Are We on the Right Track? Channel Research

Evaluation Communities Must Become More Closely Engaged - -Donors only Desk Bound Negative / Positive Change + 0 Theories of Change Results reporting Simple Eval Questions Good consultation Many field visits ++ Regular reporting Drivers based Public workshops Key factor A A Strong Steering Committee willing to arbitrate between approaches and to track the evaluation team in country(ies) Key factor B A highly structured approach with detailed phasing and willingness to communicate Resource Conscious Steering Committee 5 Engagement -- Public Uptake ++ 1 Internal & Static Conflict Analysis Channel Research 12

Specific Recommendation for the Final Guidance: Recommend Care in 3 Key Stages and Outline Unique Nature of Conflict Interventions Drawing Conclusions Outcome to Factors Optimal scenario Case Studies Conflict Analysis Sequencing Channel Research 13

Specific Recommendation for the Final Guidance: Recommend Care in 3 Key Stages and Outline Unique Nature of Conflict Interventions Drawing Conclusions Outcome to Factors Optimal scenario Case Studies Conflict Analysis Conflict analysis must precede sampling Sequencing To. R Channel Research Preparation Collection Analysis 14