The Impact of Vertical Scaling Decisions on Growth

The Impact of Vertical Scaling Decisions on Growth Projections: A Two Act Play Derek Briggs Jonathan Weeks Ed Wiley University of Colorado, Boulder Presentation at the annual meeting of the National Council of Measurement in Education. March 26, 2008. New York, NY. 1

Curtains Up PLOT SYNOPSIS • Growth models require some form of longitudinal data. • Implicit assumption that test scores have a consistent interpretation over time. • There are multiple technical decisions to make when creating a vertical score scale. • Do these decisions have a sizeable impact on growth projections? 2

ACT I: Creating Vertical Scales 3

Data • Outcome measure are Colorado Student Asessment Program (CSAP) test scores in reading. [Items: ~70 MC, 14 CR] • Longitudinal item responses for two cohorts of public and charter school students in the state of Colorado. • Each grade by year cell combination contains roughly 56, 000 students. 1, 379 unique schools. • Race/Ethnicity of Students: 64% White, 26% Hispanic, 6. 3% Black 4

Linking Design • (MC items, CR items) Unique Items • (MC items, CR items) Common Items 5

Creating a Vertical Scale Technical Decisions Psychometricians Make IRT Model Calibration Estimation 1. 1 PLM/PCM 1. Separate 1. EAP 2. 3 PLM/GPCM 2. Concurrent 2. ML 8 Defensible Vertical Scales 1. se 1 5. ce 1 2. sm 1 6. cm 1 3. se 3 7. ce 3 4. sm 3 8. cm 3 6

Separate Calibration Each oval represents a the linking of two separate item calibrations using the Stocking & Lord ICC approach. 7

Hybrid Calibration • Each oval represents a the linking of two separate item calibrations using the Stocking & Lord ICC approach. • Each rectangle represents the concurrent, multigroup calibration of the same grade level across two years. 8

ACT 2: Projections on Vertical Scales 9

Growth Model • Linear Mixed Effects Model (3 Level HLM) • Given 3 years of test score data for a student (grades 3 -5), project a scale score 3 years later (grade 8). • Score projection is a function of – two fixed effects (intercept & slope) – two student level random effects (level 2 intercept & slope) – two school level random effects (level 3 intercept & slope) 10

Fixed Effect Estimates Note: Scale Score Outcome is in Logit Units, Base Year = Grade 3 11

Comparing Growth Projections 3 PLM/GPCM &Separate 3 PLM/GPCM & Hybrid 1 PLM/PCM Note: Projection lines based solely on fixed effect estimates Grade 3 Grade 8 12

Estimated Correlation Between Random Effect Terms in HLM 13

Correlations of Student and School Slope Estimates by Vertical Scale 14

Empircal Bayes Estimates of School-Level Growth r =. 96 Standard Approach in Colorado: • Separate • 3 PLM/GPCM • EAP Switch to Hybrid calibration 15

Empircal Bayes Estimates of School-Level Growth r =. 88 Standard Approach in Colorado: • Separate • 3 PLM/GPCM • EAP Switch to Hybrid calibration & 1 PLM/GPCM 16

Empircal Bayes Estimates of School-Level Growth r =. 75 Standard Approach in Colorado: • Separate • 3 PLM/GPCM • EAP Switch to Hybrid calibration, 1 PLM/GPCM, MLE 17

Curtains Down • Vertical scales have (largely) arbitrary metrics. • Absolute interpretations of growth can deceive. – Students might appear to grow “faster” solely because of the scaling approach. – Does criterion-referencing (i. e. , standard-setting) take this into account? • A better approach might focus on changes in norm-referenced interpretations—but this conflicts with the NCLB perspective on growth. 18