International Initiative for Impact Evaluation What should be

International Initiative for Impact Evaluation What should be in a systematic review? Campbell Colloquium Chicago May 2013 Howard White, 3 ie Howard White www. 3 ieimpact. org

Effectiveness matters 30 -50% reduction in mortality Howard White www. 3 ieimpact. org

Howard White www. 3 ieimpact. org

Source: Gelb and Clark, ‘The Biometrics Revolution’, CGD, 2013 Howard White www. 3 ieimpact. org

Major policy shifts: • Private sector involvement • Poverty focus • Pro-environment Just 7 per cent of a total of 120 RE and energy projects mentioned poverty in their objectives Total around US$3 billion Howard White www. 3 ieimpact. org

Rural electrification design • 82% build infrastructure and 85% have institutional development • Tariff conditionality: 1/3 projects prior 1995, 14% thereafter • One quarter address rehabilitation or system losses • Few have explicitly pro-poor components (only two subsidize connection charge) Howard White www. 3 ieimpact. org

RET & Off grid electrification • Rapid growth: only 2 off-grid projects before 1995 and 31 after • Most (75%) include photovoltaic, usually solar home systems • Others are micro-hydro (34%) and wind power (31%) Howard White www. 3 ieimpact. org

Focus of Farmer Field Schools Howard White www. 3 ieimpact. org

Causal chain evidence matrix Links in the causal chain FFS 1 FFS 2 The rows are by intervention, not study Howard White www. 3 ieimpact. org

Evidence matrix: targeting FFS Does targeting work? Design Target groups Outcomes Targeting mechanism Who participates? Who is excluded? Explan. Drop- atory outs factors FFS 1 FFS 2 Who is targeted? How? Howard White Who participates? Who doesn’t (and why? ) www. 3 ieimpact. org

Evidence matrix: vertical synthesis Design Target groups Outcomes Tgtg mchnsm Who participates? Who is excluded? FFS 1 FFS 2 … FFS n Summary Vertical synthesis Howard White www. 3 ieimpact. org Explan. Drop- atory outs factors

Vertical integration: target groups Howard White www. 3 ieimpact. org

Vertical synthesis: Targeting mechanisms Categorical 45 Categorical 13 10 Individual/household assessment 26 20 2 Self-selection 5 Howard White 1 Individual/ household assessment 3 www. 3 ieimpact. org Selfselection 18

Evidence matrix: horizontal synthesis Design Outcomes Target Tgtg Who … groups mchnsm Dropouts Explan -atory factors Narrative summary FFS 1 FFS 2 … FFS n Summary Horizontal synthesis Howard White www. 3 ieimpact. org

Horizontal synthesis 1 Design Indonesia Outcomes Target groups Tgtg mchnsm Existing group Access to irrigation Literate High social standing Chosen by Better off community and Better educated extension Larger farms workers Howard White Narrative summary Who participates www. 3 ieimpact. org Local selection succeeded in targeting better off, educated farmers

Horizontal synthesis 2 Design Liberia Outcomes Target groups Tgtg mchnsm 50 -50 men and women Categorical Mostly men (women) Community based Howard White Explanatory factors Narrative summary Women • Lacked tools and land • Lack of time and no childcare • Need husband’s permission Targeting failed to reach women as they did not have the requisites for participation, notably time, land tools Who participates www. 3 ieimpact. org

Evidence matrix: overall synthesis Design Outcomes Target Tgtg Who … groups mchnsm Dropouts Explan -atory factors Narrative summary FFS 1 FFS 2 … FFS n Summary Overall synthesis Targeting has worked where it has targeted the farmers who are amongst the better off (as more educated, social leaders, more land, right crop etc. ) but been less successful – though there are exceptions – when targeting the poor and women who don’t have these attributes Howard White www. 3 ieimpact. org

Evidence matrix: overall synthesis Design … …. Outcomes … …. Explan -atory factors FFS 1 FFS 2 FFS 3 Sub-group 1 summary FFS 4 FFS 5 FFS 6 … FFS n Sub-group 2 summary Howard White Sub-group synthesis www. 3 ieimpact. org Narrative summary

So what goes in a systematic review? • Design and importance: Define intervention, its dimensions and design (global portfolio review) • Implementation experience • Beneficiary knowledge, perceptions and participation (access/use) • Outputs produced • Intermediate and final outcomes (effectiveness) Howard White www. 3 ieimpact. org

Rural electrification: outputs • Bangladesh: rural connections near 0 in early 80 s to more than 4 million in 2002 of which Bank 600, 000 (15%) • Indonesia: coverage rose 33 to 85% from 1993 to 2003, 11 million new h/h, of which 10 million under First and Second Indonesia RE projects (1/2 paid for by Bank) • Laos: 1993 -2004 coverage from 14 -45% Howard White www. 3 ieimpact. org

And so what about effectiveness? Grid ERR=28% Howard White Off-gird ERR=21% www. 3 ieimpact. org

Conclusions • Systematic reviewing is a method in which the approach should be systematic, but that need not be restricted to effectiveness • By addressing a broader range of policy questions SRs will be of interest to a broader range of policy makers • … And more interesting to read Thank you. Go forth and influence policy Howard White www. 3 ieimpact. org