1 IMPACT EVALUATIONS AT IFADIOE Fabrizio Felloni Deputy

1 IMPACT EVALUATIONS AT IFAD-IOE Fabrizio Felloni, Deputy Director Independent Office of Evaluation of IFAD (IOE) 19 May 2017 – Presentation to the Independent Evaluation Division, UNIDO

2 Background • IFAD Management launched in 2013 an impact assessment initiative (over 30 IAs 2013 -15), with quantitative techniques to attribute impact to project activities • IOE-IFAD requested by Executive Board to review the above initiative. • Since 2013 IOE conducted 1 impact evaluation per year to: (i) upgrade its technical skills; (ii) better engage in IFAD and external fora; and (iii) provide hands-on assessment on impact assessment initiative

3 Impact Evaluations conducted so far • Sri Lanka (2013). Quantitative survey (~2, 500 hh) + focus group discussion and follow-up technical mission • India (2014). Quantitative survey (~8, 800 hh) + focus group discussion • Mozambique. Quantitative survey (~1, 500 hh) + focus group discussion • Ongoing: Georgia: Quantitative survey (~4, 000 hh) + focus group discussion

4 Characteristics Cover all evaluation criteria • Relevance • Effectiveness • Efficiency • Impact • Sustainability • Gender equality • Innovation • Scaling Up • Natural Resource Management • Climate Change Adaptation • Performance of partners (IFAD, Government) Include both: • Quantitative and • Qualitative data collection and analysis • Primary data collection

5 General approach • Quantitative part compares between household with and without project support (treatment vs nontreatment). Focuses on whether and what changed • Qualitative part focus on understanding why (mechanisms) • Technical validation mission: covers other evaluation criteria and further validates findings

6 Main constraints • Non-existent or poor baseline surveys or missing databases ØCan not use “difference in difference” methods • Non-random selection of project beneficiary Ø Used propensity score matching (non-parametric) and Heckman selection procedure (parametric) to correct for sample selection bias Ø They do not strictly require baseline data Ø Included some recall questions in the questionnaire

7 Ideal situation (so far not found) • Compare a sample of units of observations (persons, households) with and without project. And observe the differences before and after I n c o m e with project without project Project’s contribution p c Total change 2000 Before 2006 After Time

8 Other methodological issues • Reconstruct a theory of change highlight causal chain and key assumptions • Level of analysis. Useful to foresee two questionnaires for quantitative part: (i) household level; (ii) community characteristics level Ø Helps understand village fixed effects on final outcomes / impacts • Beware of possible: (i) spill-over effects (spreading to non- treated groups); (ii) contamination effects (an external programme affecting project results) • Model specification is time consuming and in your final statistical output, you may have signs and levels of significance that you can not explain

9 Practical organization • Data collection and analysis conducted by consulting companies (national and international) • But IOE retained full leadership of evaluation (design of methodology, questionnaires, oversight of testing of field instruments, oversight of analysis, report drafting) Ø Better not to have an external company dictate main methodological choices • IOE conducted two-three missions: (i) reconnaissance; (ii) oversight of instrument testing in the field; (iii) follow-up technical mission • Budget US$ 200, 000 all inclusive

10 Practical organization • Important to retain former project manager / senior staff as key informants (check sampling strategy) • Important for consulting companies to invest on training of enumerators and quality of collection (vs. incentives to fill in many questionnaires) • ICT technology to reduce coding time and reduce risk of “fakes” (timing, GPS)

11 Documents • IFAD Manual: https: //www. ifad. org/documents/10180/bfec 198 c-62 fd-46 ffabae-285 d 0 e 0709 d 6 • Sri Lanka: https: //www. ifad. org/documents/10180/2 b 8 f 1 c 99 - 16 be-4 b 30 -969 b-d 93 f 03 ccca 41 • India: https: //www. ifad. org/evaluation/reports/impact_evaluation/ta gs/india/1063/7854528 • Mozambique: https: //www. ifad. org/evaluation/reports/impact_evaluation/ta gs/mozambique/1517/36805916