Does the Mathematics Recovery Program Improve Students Mathematics

Framing the Problem • Children enter school with a wide range of math abilities

An Early Mathematics Intervention • Math Recovery (MR) is a early arithmetic intervention that

MR Program Logic Model Tutor training in Learning and Instructional Frameworks Initial diagnostic assessment

Research Questions • Does participation in MR raise the mathematics achievement of low performing

Research Setting and Participants • Setting – 20 elementary schools (five urban, ten suburban

Multisite randomized field trial • From the pool of students identified as eligible for

Tutor selection • District coordinators recruited 18 teachers to receive training and participate as

Assessments Administered • MR 1. 1 Screening Assessment – Designed to identify where students

Figure 1: MR Treatment and Assessment Cycle for Each School MR 1. 1 =

Sample characteristics of entering 1 st graders

Data Analysis • 2 -level hierarchical linear model (HLM) with students (i) nested within

Model for assessing the effectiveness of MR at the end of 1 st grade

Interactions tested • No treatment X year effect – Justifies pooling data across years

How much do the effect sizes change if the sample is limited to students

Do the effects of participation in MR last through the end of 2 nd

WJIII Applied Problems Predicted Values— start of 1 st to the end of 2

Summary • Positive causal effects on proximal and distal outcome measures (WJIII) from participating

Tutor Fidelity: Process Perfect Fidelity No Fidelity Ongoing Assessment ZPD Nature of Instruction Positive

“Positive Infidelity” (Cordray & Hulleman, 2009) Includes instructional practices identified in mathematics education research:

What did we learn from studying Math Recovery? • First evaluation of the causal

Tutor Fidelity: Process Perfect Fidelity = expert mean (n=15) No Fidelity Ongoing Assessment* ZPD

Model for assessing the Influence of fidelity to exposure/duration of MR model (end of

Slides: 45

Download presentation

Does the Mathematics Recovery Program Improve Students' Mathematics Learning? Thomas Smith, Paul Cobb, Dale Farran, David Cordray and Charles Munter Vanderbilt University The research reported here was supported by the Institute of Education Sciences, U. S. Department of Education, through Grants R 305 B 070554 and R 305 B 040110 to Vanderbilt University. The opinions expressed are those of the authors and do not necessarily represent views of the U. S. Department of Education.

Framing the Problem • Children enter school with a wide range of math abilities – Baroody, 1987; Dowker, 1995; Gray, 1997; Griffin & Case, 1999; Housasart, 2001; Wright, 1991, 1994 a; Young-Loverage, 1989 • Children who are less ready for school typically come from low SES families, are of racial/ethnic minority backgrounds, have parents who do not speak English – Alexander & Entwistle, 1998; Berends et al, 2005; Crosnoe, 2005 • Pre-K math ability is highly predictive of achievement at the end of first grade and even at the end of 5 th grade – Duncan, Claessens, and Engel, 2004; Princiotta, Flanagan, and Germino Hauskens, 2006 • In the absence of intervention, the initial gap in mathematics achievement continues to widen – Aunola, Leskinen, and Lerkkanen, 2004

An Early Mathematics Intervention • Math Recovery (MR) is a early arithmetic intervention that involves diagnostic, one-to-one tutoring. • MR was first implemented in the United States in 1999 and has served over 3, 000 students in 19 states. • How does it work? – Tutors are typically selected from among elementary teacher in a school implementing the program (½ time release ) – Tutor Training – Pullout, one-on-one tutoring – Up to 55 half-hour tutoring sessions every day over an 11 week period – Each session is videotaped; tutors review to plan next session – Each tutor works with 9 -12 students per year

MR Program Logic Model Tutor training in Learning and Instructional Frameworks Initial diagnostic assessment Student profile assigned using MR Learning Framework MR Instructional Framework used to identify appropriate types of instructional tasks Students develop increasingly sophisticated strategies for solving number problems Tutors pose tasks within student’s zone of proximal development Students catch up to their peers and are successful in regular math classes

What does MR tutoring look like?

Research Questions • Does participation in MR raise the mathematics achievement of low performing first-grade students? • MR claim— “students performing below the 25 th percentile will benefit most from Math Recovery” • Are any gains made from participation in MR maintained through the end of second grade? • We also test the extent to which tutor knowledge and fidelity of implementation influence the effectiveness of MR.

Research Setting and Participants • Setting – 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states – Each school was a ‘fresh site’ • Participants – Students were selected for participation at the start of first grade based on kindergarten teacher recommendations and their performance on MR’s screening interview and follow-up assessment interview • Students with IEPs, speech/language support, ESL support, or excessive absences were excluded – The number of students eligible for tutoring ranged from 17 to 36 across each of the 20 schools.

Multisite randomized field trial • From the pool of students identified as eligible for MR within each school, students were randomly assigned to a tutoring cohort of 3 students each with a different start date – Cohort A—September – Cohort B—December – Cohort C—March • The remaining students assigned to the “waiting list” – selected to join an MR tutoring cohort if an assigned participant left their school or were deemed “ineligible” due to a special education placement • Process repeated in year 2

Tutor selection • District coordinators recruited 18 teachers to receive training and participate as MR tutors from the participating districts—all of whom had at least two years of classroom teaching experience. • 16 of the tutors received half-time teaching releases to serve one school each; 2 of the tutors received full-time teaching releases to serve two schools each. • All tutoring positions were underwritten by their respective school districts.

Assessments Administered • MR 1. 1 Screening Assessment – Designed to identify where students are in the MR frameworks and assess eligibility for the intervention – Assessment tasks parallel tasks used in MR tutoring • MR proximal (MRP)-- timed assessment of math fluency, designed in consultation with the program developers • Woodcock Johnson III Achievement tests (WJ III) subtests – Fluency (MF) speed of performing simple calculations for 3 minutes – Applied Problems (AP)--oral, math "word problems, ” – Quantitative Concepts (QC)-- oral questions about mathematical factual information, operations signs, etc

Figure 1: MR Treatment and Assessment Cycle for Each School MR 1. 1 = Full Math Recovery Assessment (not given by external assessor) MRp = Proximal Math Recovery Assessment (newly developed) WJ full = Full Woodcock Johnson III Achievement Assessment (Applied Problems, Math Fluency, and Quantitative Concepts) WJ MF = Math Fluency subtest of the Woodcock Johnson III Achievement Assessment only

Sample characteristics of entering 1 st graders

Data Analysis • 2 -level hierarchical linear model (HLM) with students (i) nested within tutors (j) Outcomeij=β 0 j+ β 1 j(treatment)ij+ β 2 j(pretest)ij + β 3 j(female)ij + β 4 j(lep)ij + β 3 j(frpl)ij + β 4 j(age)ij + β 5 j(Site 1)j + β 6 j(Site 3)j + β 6 j(Year 2)ij + uj+ rij Outcomes MR 1. 1 (Diagnostic Assessment) MR Proximal WJIII Math Fluency WJIII Applied Problems WJIII Quantitative Concepts WJIII Math Reasoning (AP + QC) Independent variables Treatment Pretest (1 st Principal Component of MRP, WJMF, WJAP, WJQC in fall of 1 st grade) Female Limited English Proficient (LEP) Free or Reduced Price Lunch Status (frlp) Age at pretest (months) Site 1; Site 2; Site 3 (Site 2 suppressed) Year 2 (2007=0; 2008=1)

Model for assessing the effectiveness of MR at the end of 1 st grade Assessment Effect size Sig N

Model for assessing the effectiveness of MR at the end of 1 st grade Assessment MR 1. 1 (Diagnostic Assessment) Effect size Sig 0. 85 *** N 759

Model for assessing the effectiveness of MR at the end of 1 st grade Assessment MR 1. 1 (Diagnostic Assessment) MR Proximal Effect size Sig 0. 85 *** 0. 26 *** N 759 775

Model for assessing the effectiveness of MR at the end of 1 st grade Assessment MR 1. 1 (Diagnostic Assessment) MR Proximal WJIII Math Fluency Effect size 0. 85 0. 26 0. 14 Sig *** * N 759 775

Model for assessing the effectiveness of MR at the end of 1 st grade Effect Assessment size MR 1. 1 (Diagnostic Assessment) 0. 85 MR Proximal 0. 26 WJIII Math Fluency 0. 14 WJIII Applied Problems 0. 25 WJIII Quantitative Concepts 0. 21 WJIII Math Reasoning (AP + QC) 0. 26 Sig *** *** *** N 759 775 775 775

Interactions tested • No treatment X year effect – Justifies pooling data across years – Implies tutors are not more effective in their second year • No treatment by site effect – No evidence that tutors in some sites were more effective at increasing student learning than in other sites • Treatment X pretest effect for some assessments – Low performing 1 st grade entrants appear to benefit more from math recovery than their higher performing counterparts in mathematical reasoning

How much do the effect sizes change if the sample is limited to students performing below the 25 th percentile at the start of 1 st grade? Full Sample Assessment MR 1. 1 (Screening) MR Proximal WJIII Math Fluency WJIII Applied Problems WJIII Quantitative Concepts WJIII Math Reasoning (AP + QC) ES 0. 85 *** 0. 26 *** 0. 14 * 0. 25 *** 0. 21 *** 0. 26 *** * p<0. 05, ** p<0. 01, *** p<0. 001 (two tailed tests) Below 25 th%ile on WJIII MR Pretest ES % difference

Do the effects of participation in MR last through the end of 2 nd grade? Full Sample Assessment MR Proximal (2 nd grade) WJIII Math Fluency WJIII Applied Problems WJIII Quantitative Concepts WJIII Math Reasoning (AP + QC) † P<=. 10 (one tailed-test) ES Sig Below 25 th%ile on WJIII MR Pretest N ES N

Do the effects of participation in MR last through the end of 2 nd grade? Full Sample Assessment MR Proximal (2 nd grade) ES Sig Below 25 th%ile on WJIII MR Pretest N 0. 06 NS 320 -0. 02 NS 320 WJIII Applied Problems 0. 08 NS 320 WJIII Quantitative Concepts 0. 00 NS 320 WJIII Math Reasoning (AP + QC) 0. 05 NS 320 WJIII Math Fluency † P<=. 10 (one tailed-test) ES N

Do the effects of participation in MR last through the end of 2 nd grade? Full Sample Assessment MR Proximal (2 nd grade) ES Sig Below 25 th%ile on WJIII MR Pretest N ES N 0. 06 NS 320 0. 13 NS 105 -0. 02 NS 320 0. 30 † 105 WJIII Applied Problems 0. 08 NS 320 0. 23 † 105 WJIII Quantitative Concepts 0. 00 NS 320 0. 00 NS 105 WJIII Math Reasoning (AP + QC) 0. 05 NS 320 0. 14 NS 105 WJIII Math Fluency † P<=. 10 (one tailed-test)

WJIII Applied Problems Predicted Values— start of 1 st to the end of 2 nd grade

Summary • Positive causal effects on proximal and distal outcome measures (WJIII) from participating in MR at the end of first grade – Some evidence MR is more effective for lowest performing students • Positive effects fade by the end of second grade – Some evidence of long term increases in math fluency and applied problems among participants who started 1 st grade performing below the 25% percentile • We cannot attribute the lack of sustained gains through the second year solely to poor implementation – Process components: fairly faithful to the MR model – Structural components: less faithful to the MR model

Tutor Fidelity: Exposure/Duration

Tutor Fidelity: Adherence Perfect Fidelity No Fidelity Assessment 1. 1 Assessment 2. 1 Profile Assigned

Tutor Fidelity: Process Perfect Fidelity No Fidelity Ongoing Assessment ZPD Nature of Instruction Positive Infidelity

“Positive Infidelity” (Cordray & Hulleman, 2009) Includes instructional practices identified in mathematics education research: • Re-voicing student thinking to highlight particular mathematical ideas, to introduce mathematics vocabulary or to position students in relation to each other and their arguments (Franke, Kazemi, & Battey, 2007; O’Connor & Michaels, 1993) • Asking the student to solve a task (s)he has just solved in a different way, so that the student has an opportunity to approach the same problem from a potentially different mathematical perspective or to represent the mathematics in a different way to find methods that enable progress (Carpenter & Lehrer, 1999: NCTM, 2000) • Asking the student to compare alternative strategies and why they work, to encourage the student to be reflective and to provide opportunities to make connections between various strategies and mathematical ideas (Carpenter & Lehrer, 1999; Rittle-Johnson & Star; 2007).

What did we learn from studying Math Recovery? • First evaluation of the causal effects of MR – Tutoring meets goals immediately following treatment, but the effects fade • Why? – Tutor knowledge? • Although tutor knowledge improved among those that started low, no evidence that this improved outcomes – Fidelity of implementation? • Tutors were faithful when doing engaging in strategy based activities • They did not do this very much • Highlights the importance of attending to the developer’s Theory of Action in designing an evaluation

MR intervention components Initial Assessments Administered correctly Profile assigned “STRUCTURE” • Exposure/duration • Adherence 12 -15 weeks, 25 -30 min Ongoing assessment Instruction targets ZPD Socratic nature of instruction “Positive infidelity” “PROCESS” • Quality of Delivery • Participant responsiveness • Program differentiation

Tutor Fidelity: Process Perfect Fidelity = expert mean (n=15) No Fidelity Ongoing Assessment* ZPD Nature of Instruction Positive Infidelity *Significant (p<. 05) differences between tutors and experts

Model for assessing the Influence of fidelity to exposure/duration of MR model (end of grade 1) Outcomeij=β 0 j+ β 1 j(pretest)ij+ β 2 j(lowses)ij + β 3 j(District 1)ij + β 4 j(District 2)ij + β 5 j(Year 2)ij + + β 6 j(# of sessions)ij + β 7 j(avg session time)ij + β 8 j(avg strategy time)ij + rij β / sd(DV) ***p <. 001 **p <. 01 *p <. 05