TheoryBased Evaluation in Kenya Using Research to Inform

Theory-Based Evaluation in Kenya: Using Research to Inform National Scale Implementation Abdi Habat Director, Basic Education Kenya Ministry of Education Dr. Benjamin Piper Senior Director, Africa Education, RTI International, Research Triangle Park, North Carolina, USA

Tusome National Literacy Programme Metric TUSOME SCALE Teachers 75, 000+ Head teachers 23, 800 Coaches (CSOs) 1270 Pupils (yearly) 3. 6 million Books 19. 1 million

Testing Elements of the Tusome Causal Chain • • Are teachers trying it? Do teachers think it works? Do coaches visit classrooms? Do initial results (1 month in) show positive impact? 3

Are teachers trying it? • Are teachers trying it? – Which day and week are they on? – Review TGs and exercise books – Classroom data 4

5

Do teachers think it works? • Do teachers think it works? – Teacher survey – Teacher feedback from coaches – Response from unions 6

Qualitative Responses • “What I think about the program is that it is a very good program because it is going to help the pupils to know how to read early. When they attain reading competences early they will also be able to perform better even in all other subjects because reading is a prerequisite to learning” (TAC tutor, Embu, rural zone). • “Tusome is making pupils read better. When I test pupils, for example, by asking one of them to read, it is encouraging to see pupils read just like we do during our lessons. As I teach, I ask different pupils to read certain text and this allows me to gauge their reading ability” (Teacher, Isiolo, peri-urban school). 7

Qualitative Responses 2 • “They are learning better than before, because I have been a teacher this is my 28 th year and Tusome is coming up very well than the way we used to teach or than the way I used to teach so Tusome has made work easier than before. The system as we are reading it has sounds you make words from the sounds but before we used to teach them to learn a word without having taught them where the word has come from so after we read the sounds, we blend the sounds we make the word. It is very interesting. Therefore, the pupils learn how to read without much struggle” (Teacher, Kiambu, peri-urban school). 8

Do coaches visit classrooms? • Do coaches visit classrooms? – Tusome observation data – 20, 000+ observations per month 9

Number of coaching visits per month Coach dashboard presented Thursday Dr. Evelyn Jepkemei 10

Are initial results positive? 11

Tusome Class 2 Learning Outcomes 70 60 Tusome Impact Data Presented Tomorrow by Tusome National Coordinator 60 Esther Kinyanjui 55 50 40 37 37 30 37 37 27 26 17 20 11 23 22 41 40 36 24 24 18 14 12 10 0 English Kiswahili English Kiswahili Letter Sound Fluency Oral(cpm) Reading Fluency Reading (cpm)Comprehension Emergent % Benchmark %Zero Score % 42186 42552

Tusome Class 2 Effect Sizes 0. 60 0. 55 0. 50 0. 44 0. 40 0. 30 0. 20 Tusome Impact Data Presented Tomorrow by Tusome National Coordinator Esther Kinyanjui 0. 10 0. 00 English Kiswahili

Testing Elements of the Causal Chain • • Are teachers trying it? YES Do teachers think it works? YES Do coaches visit classrooms? YES Do initial results (1 month in) show positive impact? YES 14

Using RCTs to Design Tusome 15

RCTs that Preceded TUSOME • • • Coach to teacher ratios? Literacy and numeracy programs together? Mother tongue programs effective in other subjects? Which ingredients matter most? – Training and coaching – New book at 1: 1 ratio – Teachers’ guide • Which ICT program can support learning? – E-readers for kids – Tablets for teachers – Tablets for coaches 16

Coach to Teacher Ratios Randomly assigned coaches • 10: 1 outperformed 15: 1 • 15: 1 more cost-effective • Nonformal schools issues Coach visits • No random assignment • Similar findings Key: Use 15: 1 ratios 17

Impact on Literacy and Numeracy • 0. 2 to 1. 0 SD effect size on literacy • 0. 1 to 0. 4 SD effect on numeracy • Tested in peri-urban and rural areas • Literacy scaled up in Tusome • GPE funded national scaleup in 2016 Key: Subject interactions? 18

Can Mother Tongue Interventions Work in Kenya? • Medium scale mother tongue literacy study • Compared mother tongue with non-mother tongue • 0. 2 to 0. 7 SD • Decoding vs. comprehension • 2 different languages • Language homogenous • Implementation complexity Key: Yes, MT can work 19

Do Mother tongue programs affect other subjects? • Comparing mother tongue and non-mother tongue • Outcome was other subjects • Most important question for communities on MT • Study for only a year • Not MT as LOI • Very hard to manage • Controversial results Key: MT didn’t help? 20

Which ingredients matter most? • Which ingredients? – Training & Support – Books – Training Guides • • • Books matter Teachers’ guides matter Large effects: 0. 1 to 1. 0 SD TGs are cost effective Include both Key: TGs really help 21

Which ICT program best supports learning? Pupil e-reader vs. teacher tablet vs. coach tablet • • • E-reader costs are high 2 zones of 10 schools Rural and peri-urban 0. 2 to 0. 6 SD effect size Coach tablet is most costeffective as well Key: Use coach tablets for cost-effectiveness 22

Tusome Theory-based Evaluation Results • • • Coach to teacher ratios? 15: 1 is better Literacy and numeracy programs together? Yes Mother tongue programs effective in MT? Yes Mother tongue programs effective in other subjects? No Which ingredients matter most? – Training and coaching ! – New book at 1: 1 ratio !!! – Teachers’ guide !!!!! • Which ICT program can support learning? – E-readers for kids Yes $$$$ – Tablets for teachers Yes $$ – Tablets for coaches Yes $ • Tusome included almost all of these elements 23

Implications • • • RCTs should answer local policy questions RCTs should test mechanisms (Deaton, 2010) Include working through civil servants RCTs should help scale-up Use theory-based studies to test leading indicators – Classroom utilization – Estimate buy-in – Fail fast, but with data • Monitor what matters 24

Thank you bpiper@rti. org 25

Reasons for Failure • • • Teachers never try it Teacher fall at the first hurdle Feels like extra work It doesn’t work Mid-level civil servants uninterested Program is “on the side” – What drives instruction? • Slow reaction time to challenges 26

Thomas Guskey change model Training Understanding Change Old model Understanding 80% Training 40% Evaluating 15% Trying Guskey model 5% Change

Do coaches think it works? Coach Survey Questionnaire item The tablets help me have a better understanding of literacy instruction Agree or Strongly Agree 98. 2 Tablet has improved my ability to be a coach 98. 3 The tablets help me to visit more teachers regularly 91. 0 Tablet helps me feel more alert and engaged during lesson observation 98. 5 Software facilitates my ability to have engaging feedback sessions 98. 1 The tablets help me know what/how to observe when I visit a classroom 98. 4 Tablet makes it easier to demonstrate appropriate teaching 97. 0 28

29

Kenya pilot studies RCT Studies PRIMR midterm (Piper, Zuilkowski & Mugenda, 2014) Subjects Did it Work? English, Kiswahili Yes PRIMR endline (Piper, Jepkemei & Kibukho, 2015) English, Kiswahili Yes PRIMR endline (Piper, Ralaingita, Akach & King, 2016) Mathematics Yes Eng, Kisw, Math Mostly Eng, Kisw, Math No English, Kiswahili Mostly PRIMR Coaches 10: 1 or 15: 1 (Piper & Zuilkowski, 2015) PRIMR Coach visits (Piper & Zuilkowski, 2015) E-reader, tch tablet, coach tablet (Piper, Jepkemei, Kwayumba & Kibukhko, 2015) Mother tongue (Piper, Zuilkowski & Ong’ele, 2016) 2 mother tongues Mostly Mother tongue on other subjects Eng, Kisw, Math No Training vs. Training+Books vs. Training/Books+Teachers’ Guide English, Kiswahili, Some. Math what 30

Tusome Mini-EGRA: Is it Working? Implementer Date Counties Schools Children MSI (external) July 2015 26+APBET 204 4866 Tusome & GOK July 2016 26+APBET 204 3992

Tusome Data Dashboard Tusome classroom observation dashboard 32