Sensitivity of Teacher ValueAdded Estimates to Student and

Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables October 2013 Matthew Johnson �Stephen Lipscomb �Brian Gill

Value-Added Models (VAMs) Used Today Differ in Their Specifications Value-Added Model Multiple Student Peer Years of Characteristics Prior Scores Chicago Public Schools Yes No No DC IMPACT Yes No Florida Yes Yes Pittsburgh Public Schools Yes No SAS EVAAS No No Yes 2

Research Questions § How sensitive are teacher VAM estimates to choice of control variables? – Are estimates for teachers with more students from disadvantaged backgrounds affected by this choice? § Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates? § Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates? 3

Data § Data from a northern state and a medium-sized urban district in that state – District has more minority and low-income students than state average § Estimate separate VAMs using state data and district data – More control variables available in district VAMs – For peer characteristics, use teacher-year level averages in state VAMs, classroom averages in district VAMs § Each VAM uses three years of teaching data from 2008 -2009 through 2010 -2011 4

Baseline Model § Explore sensitivity to several specifications: – Exclude peer average characteristics (X i, t) – Exclude student characteristics (Xi, t) and peer characteristics (X i, t) – Add scores from two prior years (Yi, t-2) – Interact free/reduced lunch status with baseline scores § Estimate all models using the same set of student observations § Control for measurement error in prior test scores using an errors-in-variables approach § Empirical Bayes (shrinkage) adjusted estimates 5

Student and Peer Characteristics Student Controls (State) Student Controls (District) Peer Averages (State) Peer Averages (District) Free or Reduced-Price Meals x x Disability x x Race/Ethnicity x x Gender x x English Language Learner x x Age/Behind Grade Level x x Gifted Program Participation x x Lagged Rate of Attendance x x Lagged Fraction of Year Suspended x x Average Lagged Achievement x x SD of Lagged Achievement x x Number of Students in Class x 6

Correlation of 8 th-Grade State Teacher VAM Estimates Relative to Baseline Specification Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Math (N = 2, 778) Reading (N = 3, 347) Exclude peer characteristics 0. 970 0. 982 Exclude student and peer characteristics 0. 964 0. 979 Add scores from t-2 0. 977 0. 958 Add scores from t-2 and exclude student/peer characteristics 0. 946 Findings are based on VAM estimates from 2008– 2009 to 2010– 2011 on the sample of students. 7

Percentage of 8 th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification Remove Student/Peer Controls and Add t-2 Scores Baseline Model 1 st (Lowest) 2 nd 3 rd 4 th 5 th (Highest) 1 st (Lowest) 81 17 1 1 0 2 nd 18 57 23 3 0 3 rd 1 23 53 22 1 4 th 0 3 22 59 16 5 th (Highest) 0 0 1 16 83 Findings are based on VAM estimates for 3, 347 reading teachers in grade 8 from 2008– 2009 to 2010– 2011. Correlation with baseline = 0. 946. 8

How Are Teachers in One District Affected? § District has relatively large fraction poor and minority students Percentile Rank of District Teachers in State Distribution Math Grade 8 District Percentile Rank Reading Grade 8 15 50 85 Baseline 21 62 86 17 66 90 Exclude peer characteristics 23 59 86 13 59 87 Exclude student and peer characteristics 20 58 85 13 57 89 State Percentile Rank: 9

Using Additional Controls in District Data Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Math Grades 6 -8 (N = 164) Reading Grades 6 -8 (N = 215) Exclude peer characteristics 0. 955 0. 963 Exclude student and peer characteristics 0. 918 0. 949 Add scores from t-2 0. 987 0. 967 Add scores from t-2 and exclude student/peer characteristics 0. 927 0. 909 Findings are based on VAM estimates from 2008– 2009 to 2010– 2011 on the sample of students. 10

Teacher-Year Average Student Characteristics vs. Classroom Average Math Grades 6 -8 (N = 164) Reading Grades 6 -8 (N = 215) Correlation Between Effect Estimates 0. 956 0. 975 Average Standard Error (Classroom) 0. 065 0. 073 Average Standard Error (Teacher) 0. 069 0. 075 11

Different Relationship Current and Prior Test Scores for FRL Students Math Grade 8 (N = 2, 778) Reading Grade 8 (N = 3, 347) Non-FRL Student Coefficient on Prior Year Math Score (SE) 0. 847 (0. 002) 0. 199 (0. 003) FRL student coefficient on Prior Year Math Score (SE) 0. 896 (0. 003) 0. 154 (0. 004) Non-FRL Student Coefficient on Prior Year Reading Score (SE) 0. 086 (0. 002) 0. 641 (0. 003) FRL student coefficient on Prior Year Reading Score (SE) 0. 031 (0. 003) 0. 728 (0. 004) § Correlation of teacher effect estimates with baseline model above 0. 99 for both subjects 12

Conclusions § Teacher VAM estimates highly correlated across specifications – Choice of control variables – Use of teacher-year level averages in place of classroom averages – Interaction between FRL status and prior scores § Choice of control variables can impact estimates for teachers of disadvantaged students 13

Context for Results § Other researchers have examined correlations in teacher effect estimates when different same -subject assessments are used as outcomes for teacher VAMs § The highest correlations these authors found are: – – – Lockwood et al. (2007): 0. 46 Sass (2008): 0. 48 Concoran et al. (2011): 0. 62 Lipscomb et al. (2010): 0. 61 Papay (2011): 0. 54 14

For More Information § Please contact – Matthew Johnson • MJohnson@mathematica-mpr. com – Stephen Lipscomb • SLipscomb@mathematica-mpr. com – Brian Gill • BGill@mathematica-mpr. com 15 Mathematica® is a registered trademark of Mathematica Policy Research.