Scalingup education reforms in Kenya An evaluation of

Scaling-up education reforms in Kenya: An evaluation of the nationwide Teacher Internship Programme Tessa Bold, IIES Stockholm and Goethe University Frankfurt Mwangi Kimenyi, Brookings Institution Germano Mwabu, University of Nairobi Justin Sandefur, Center for Global Development Alice Ng’ang’a, Strathmore University December 2011

Outline • • Introduction Randomization Results Conclusions

Context: Kenya's primary education system Free Primary Education enacted in 2003: • Abolished fees in all government schools, ending fund-raising role of PTA • FPE increased enrollment in government schools, but at the same time unexpected increase in private schools enrollment • Private school market share trebled in last decade; massive performance deferential.

Context: Kenya's primary education system Focus of the study is government schools: • Education in Kenya is highly centralized under MOE; limited local gov't role • Unionized teachers employed directly by Teacher Service Commission (TSC) • Other school expenses funded by a $10/pupil annual gov't grant to school bank account • Corruption in use of FPE funds leading to Aid freeze.

Context: Teacher hiring TSC teachers • Average earnings approximately $250/month • National pupil-teacher ratio approximately 40: 1, but acute shortages in some regions • Long queue for TSC employment. Excess supply of trained teachers • Teachers in queue often teach on contract or in private sector.

Context: Teacher hiring Contract and PTA teachers • Employed informally out of (unofficial) fee revenue • Average earnings roughly 1/5 of TSC teachers • Majority have P 1 qualifications (requirement for intervention described here)

Context: Teacher hiring TSC Teachers PTA Teachers Terms of Employment Recruitment Permanent Informal Contract District School Employment Nairobi School Salary $257/mo $55/mo. Share 82. 5% 17. 5%

Genesis of this project Ministry of Education • Desire to formalize the system of contract teachers, fill vacancies at lower cost • Research pilot as means to avoid legal battles with Union Project researchers • Goal: Work with MOE to integrate rigorous evaluation in policy formulation • Enhance capacity of MOE to evaluate their policies.

Starting Point: Recent RCT Literature Growing body of evidence on importance of teacher contracts & incentives: • Banerjee, Cole, Duo, & Linden (2007): Balsakhi tutors in Mumbai targeting lagging students raised overall performance by 0. 28 std. dev. • Muralidharan & Sundararaman (2008): Contract teachers in Andhra Pradesh raised performance by 0. 15 std. dev. , comparable to (non-experimental measure of) benefits of extra civil service teachers.

Recent RCT Literature Kenyan Literature • MOE Report on an RCT designed to raise pupil test performance in Western Kenya • Class size: Massive reduction, from 82 to 43 pupils ) no effect on scores • Contract teachers: Produced scores 0. 21 std. dev. higher than TSC teachers. • SMC training: Substitute for contract teachers: some positive effect, but not for pupils exposed to contract teacher.

Motivation: Challenges of Scaling Up • RCT evidence from Western Kenya suggests contract teachers have a significant, positive effect on performance at low cost. • We focus here on an experiment designed to identify the obstacles to scaling-up this success. 1. Geographic heterogeneity • Policymakers tend to have less faith in external validity than experimentalists! • Can success in Busia be replicated in poorer areas with a thin labor market for teachers? Or in Nairobi with higher costs of living?

Motivation: Challenges of Scaling Up 2. Institutional capacity • RCTs typically run with international NGOs, subject to intense researcher supervision. • Going to scale means working with government.

The intervention: Contract teachers • The core intervention is the provision of 1 extra teacher in 128 treatment schools • Randomly assigned to grade 2 or 3 in June 2010 (all in grade 3 in 2011) • Instructions not to reassign incumbent teacher (split class) • Treatment = reduced class size + change in contract structure • Experimental variation in contract used to unpack effect of extra teacher • Central vs. local hiring • High vs. low salary

The intervention: SMC training School Management Committees • Comprised of head teacher, teachers, parent & community reps • Exercise oversight of central gov't grant monies • Unclear authority vis-a-vis centrally hired teachers Training treatment • Implemented in 1/2 of teacher treatment sample; Head teacher + 1 parent representative invited to 1 day training • Curriculum focused on SMC's authorities to monitor teachers

Cross-cuts No of schools Treatment Control 128 64

Cross-cuts: SMC training Within treatment schools SMC training No. of schools 64 64

Cross-cuts: NGO vs. Government Within treatment schools SMC training NGO schools 32 No SMC training 32 Government schools 32 32

Randomization set up • Use an optimal multivariate matching algorithm to achieve a balanced randomization (see Greevy et al. (2004) and Bruhn and Mc. Kenzie (2008)) • Treatment and control schools were matched along the following dimensions: Ø Results in nationwide end-of-primary leaving exams, results in Grade 1 baseline test; enrolment; no. of classrooms; no. of TSC teachers; no. of contract teachers and average pay of contract teachers.

Checking balance: Specification • We test the outcome of interest for pupil i in school j in period t, • Let be the treatment in school j • To examine whether the treatment and control schools are comparable prior to the intervention, we estimate The treatment of interest is whether a school received a teacher or not. • There are two types of outcome variables: • Variables that were used in matching schools prior to randomization (all collapsed at the school level) • Additional test score information collected at baseline that was not used in conducting the randomization.

Checking balance: School characteristics Table: Difference in School Characteristics between Treatment and Control Schools Control Treatment Difference Enrolment 43. 33 53. 26 9. 935 (7. 418) No. of classrooms 11. 76 12. 48 . 715 (1. 046) No. of civil service teachers 10. 02 10. 21 . 195 (1. 002) No. of contract teachers 1. 90 2. 27 . 369 (. 347) Average pay for contract teacher 2843 3393 550. 103 (531. 535) Variables in the table were used in the matching algorithm. Regressions based on 161 schools. Standard errors in brackets.

Checking balance: Learning outcomes Table: Difference in Test Scores between Treatment and Control Schools Control Treatment Difference KCPE 239. 48 235. 083 Grade 1 English Grade 1 Maths . 028 . 074 . 060 . 063 -4. 396 (6. 783). 046 (. 166). 003 (. 156) Variables in the table were used in the matching algorithm. Regressions based on 161 schools. Standard errors in brackets.

Checking balance: additional checks Table: Difference in Test Scores between Treatment and Control Schools Grades 2 and 3 Total English Math (1) (2) (3) . 132 (. 489) . 015 (. 887) . 117 (. 230) Regressions based on 4187 students in 155 schools. Standard errors clustered at school. P-values reported in brackets.

Results The Treatment Effect: Specification • Denote by the outcome of interest for pupil i in school j in period t • Let be the treatment in school j • To examine whether the effect of the treatment following the intervention, we estimate • The treatment of interest is whether a school received a teacher or not. • The outcome variable of interest is measured as performance on English and Mathematics tests in Grade 3 and 4.

Results: Adding a teacher Table: Effect of an additional teacher on learning outcomes All grades Grade 3 Grade 4 Total English Math (1). 297 (. 078)*. 271 (. 130). 338 (. 086)* (2). 172 (. 065)*. 138 (. 157). 223 (. 044)* (3). 125 (. 128). 132 (. 141). 115 (. 241) Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level. P-values reported in brackets. The effect is equivalent to a 0. 16 std increase in test score

Results: Comparing Government and NGO Table: Comparing the effect of an additional teacher on learning outcomes in government and NGO schools Teacher x Government Total English Math (1) (2) (3) . 235 (. 198) . 150 (. 110) . 084 (. 096) . 195 (. 113)* . 167 (. 102)* Teacher x NGO. 362 (. 208)* Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level reported in brackets. Both government and NGO show positive effect, but only significant for NGO. For NGO implementation, adding an additional teacher raised test scores by 0. 2 standard deviations.

Results: SMC training Table: Examining the effect of SMC training on teacher effectiveness Teacher x no SMC training Total (1). 208 (. 185) English (2). 152 (. 102) Math (3). 055 (. 091) Teacher x SMC training . 398 (. 223)* . 194 (. 122) . 203 (. 107)* Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level reported in brackets. Combining SMC training and an additional teacher raised test scores by 0. 22 standard deviations.

Results: Salary variation Table: Examining the effect of salary level on teacher effectiveness Total English Math (1) (2) (3) High Salaried Teacher . 376 (. 301) . 212 (. 166) . 164 (. 142) Low Salaried Teacher . 272 (. 174) . 159 (. 096)* . 113 (. 086) Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level reported in brackets. High salaried teachers increase test scores more, but difference is not significant.

Results: Type of employment contract Table: Examining the effect of employment contract on teacher effectiveness Total English Math (1) (2) (3) Centrally. 262 employed teacher (. 208) . 146 (. 114) . 117 (. 100) Locally employed teacher . 203 (. 107)* . 135 (. 098) . 338 (. 196)* Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level reported in brackets. Locally employed teachers increase test scores by 0. 19 standard deviation.

Results: TSC Teachers versus Contract Teachers Table: Comparing civil service teachers and community teachers Total English Math (1) (2) (3) -. 349 (. 381) -. 191 (. 221) -. 158 (. 173) Community contract -. 304 teacher (. 236) -. 178 (. 127) -. 126 (. 117) Teacher . 265 (. 110)** . 193 (. 103)* TSC teacher . 458 (. 205)** Regressions based on 7613 pupils in 162 schools. Standard errors clustered at school level reported in brackets. No significant difference between TSC teachers and PTA teachers.

Conclusions Summary of findings • Positive impact of reducing class size and of similar quantity as existing literature. • Positive interaction between additional teacher and SMC training. • Both government and NGO administered schools show positive effect, but significant only for NGO. • No evidence that TSC teachers do better. Contract teachers achieve same outcomes at a fraction of the cost.

THANK YOU