Successful partnerships for evidencebased policymaking Lessons from JPAL
Successful partnerships for evidence-based policy-making: Lessons from J-PAL Evidence on Conditional Cash Transfers in Education Laura Poswell, J-PAL Africa IGC Workshop Maputo 6 July 2015
Overview 2 Who is J-PAL Measuring impact: Why randomise Evidence from randomized evaluations on conditional cash transfers for education � Mexico (PROGRESA) � Colombia � Malawi Lessons and nuances
Overview 3 Who is J-PAL Measuring impact: Why randomise Evidence from randomized evaluations on conditional cash transfers for education � Mexico (PROGRESA) � Colombia � Malawi Lessons and nuances
4
J-PAL: Over 110 affiliated researchers and 600 projects in 64 countries
J-PAL’s Work in Education 6 165 completed and ongoing evaluations in 36 countries
Overview 7 Who is J-PAL Measuring impact: Why randomise Evidence from randomized evaluations on conditional cash transfers for education � Mexico (PROGRESA) � Colombia � Malawi Lessons and nuances
How should we increase secondary school participation? 8 Address health issues (malnutrition, intestinal parasites? ) Lower the costs of schooling to families? Inform parents of the returns to secondary schooling?
Attendance How to measure impact Intervention Impact Counterfactual Time 9
How to measure impact Counterfactual Attendance Impact Intervention Time
Counterfactual 11 Counterfactual: Would have happened in the absence of the policy? Problem: Will never observe what would have happened How to measure the causal impact of a policy?
Randomised Evaluations in practice 3. Intervention 4. Endline 1. Baseline 2. Random assignment 3. Comparison
Key advantage of randomised evaluations Members of the groups are statistically identical thus any change can be attributed to the program itself.
14 Partnerships for randomised evaluations Implementing Partner (Government, NGOs, Business) Research team (Academics, fieldwork team) Question – Intervention + Evaluation – Results – Scale up Start early together
Overview 15 Who is J-PAL Measuring impact: Why randomise Evidence from randomized evaluations on conditional cash transfers for education � Mexico (PROGRESA) � Colombia � Malawi Lessons and nuances
Conditional Cash Transfers 16
Evidence from randomised evaluations on CCTs in education 17 Conditional cash transfer (CCT) programs widely tested, consistently effective at increasing participation � Over 30 countries have some form of CCT Program Impacts on learning less clear Differential impacts? � More or less effective for the most marginal? Other impacts beyond education? Can careful design improve effectiveness?
18 PROGRESA : landmark program and evaluation Context of Program: Beginning of Mexico’s Economic Crisis in 1994 � Poor and marginalised populations lagged behind in terms of education, health, and nutrition. Group of cabinet officials with presidential support want to introduce conditional cash transfers for the poorest. � Faced some objections Pilot and then a rigorous evaluation
PROGRESA 19 PROGRESA program details Included education, nutrition, and health conditions. � Two-stage targeting mechanism (geographic, household) � Ave. transfer amount = 22% of the monthly income of beneficiary families. � Evaluation integrated into programme design Evaluation at scale planned from the outset � Budget and administration constraints � Randomised phase-in design (320 localities receive program in first 2 years, remaining 186 in year 3) �
PROGRESA evaluation outcomes 20 Education outcomes Increased enrolment rates especially for children transitioning to junior secondary school (11. 1 percentage points) And especially for girls (14. 8 percentage points) Lower dropout rates Reduced child labour � No significant impacts on learning outcomes References: Shultz (2001), Behrman et al 2001)
PROGRESA-OPORTUNIDADES 21 Implications of Rigorous Evaluation � Credibility People believed the results – international recognition Made it politically robust – new government in 2000 continues and expands the program 2. 6 million families by 2000 � Precedence Set a new high standard for the design and conduct of social policy - early involvement of government & researchers Evaluation recommendations lead to new questions for further program improvements CCTs introduced in more and more countries and many rigorous evaluations done
Columbia Study Timing and size of payout 22 Structure of most CCT programs is surprisingly similar: Families receive money (often every 2 months) when their children meet specified monthly attendance targets � Usually +- 80% � In 2005 the Secretary of Education of the City of Bogota decided to pilot a new CCT programme to: Prevent drop out from secondary school � Encourage matric graduation & tertiary enrollment � Pilot in 2 of 12 localities in the city for 1 year Results to inform the design for city-wide program References: Barrera-Osorio et al. (2011)
Columbia study: 3 programs tested 23 Intervention 1 Standard CCT 2 Savings CCT Regular Transfers 30, 000 pesos (US$15)/month 20, 000 pesos (US$10)/month 3 Graduation CCT 20, 000 pesos (US$10)/month Conditions 80 % attendance /month Graduated from secondary school Additional Transfers 100, 000 pesos (US$50) at enrollment time of next school year 600, 000 pesos (US$300) immediate payment with proof of enrolment in higher education; otherwise, payment delayed by one year
Colombia study: Attendance 24 Despite reducing the amount for regular payments, the saving and graduation CCTs attendance >= standard CCT
Colombia study: Re-enrollment 25 Savings & graduation CCTs on re-enrolment > standard CCT
26 Colombia study: Tertiary enrollment Savings & graduation CCTs tertiary enrollment but standard CCT does not (not statistically significant)
Columbia study: Other effects? 27 Savings CCT � Especially effective at improving re-enrollment of the poorest students & those most at risk of dropping out � +- 12 percentage point Eligibility rules & Unintended consequences: � Families on average did not enrol all eligible children � And if they were registered, but not selected, they attended school less than a sibling who was selected
Transfer size? 28 In Colombia – the savings CCT reduced the monthly transfer by 30% from the standard CCT. Did not result in lower monthly attendance than the standard CCT study in Malawi Randomly varied CCT amounts in Malawi: $1/month to girls as effective as $2 -5, similar results for transfers to parents
Conditional vs Unconditional? 29 Malawi study – first study to test CCT vs UCT Considers effects on education and sexual behaviour of adolescent girls Zomba district, 2007 - 2009 Sample size – nearly 3, 000 schoolgirls aged 13 – 22 CCT - 80 % attendance /month UCT – no conditions References: Baird et al (2011)
Malawi study – Education Results 30 Enrollment � UCTs - 5% higher than control � CCTs – 11% higher Attendance � UCTs – no significant impact � CCTs - 8% points higher than control Learning (English, Maths, Cognitive tests) � UCTs – no significant difference � CCTs – significant improvements in all
Malawi study – Other Results Marriage/ Pregnancy 31 Married after 2 years Control: 18% married � UCT: 10% � CCT: no significant difference � Pregnant during the program Control: 25% � UCT: 18% � CCT: no significant difference � UCT effect appears to be driven by cash to school drop outs as opposed to CCT where no cash if drop out
32 Mixed Results on Importance of Conditionality In rural Burkina Faso CCT vs UCT study for children 7 – 15 , Very low enrollment - +- 50% UCTS = CCTs in erollment for non-marginal children (boys, older & higher ability children) CCTS > enrollment for marginal children (girls, younger children, lower ability children) “Labeled” cash transfers as effective as CCTs in Morocco References: Akresh et al (2013), Benhassine et al (2014)
Overview 33 Measuring impact: Why randomise Who is J-PAL Evidence from randomized evaluations on conditional cash transfers for education � Mexico (PROGRESA) � Colombia � Malawi Lessons and nuances
Lots of lessons, many nuances 34 Many are supportive Some conflict When thinking through your own design: � Try to understand the evidence � And what might be most applicable to your context � Ask new questions important to you based on work already done Work with partners from early on
35
J-PAL’s Post Primary Education Initiative 36 To promote policy-relevant research on secondary, tertiary, and vocational education in developing countries Step 1: Review paper � Highlighted gaps in the literature � Identified policies that should be given the highest priority for future research
37 J-PAL’s Post Primary Education Initiative Research questions � Pedagogy � The role of information communication technology � The design of teacher incentives � Strategies to include and support disadvantaged students � The role of private schools � Mechanisms to increase the demand for schooling among students and their parents (CCTs, information interventions)
- Slides: 37