Rerandomization to Improve Baseline Balance in Educational Experiments

Rerandomization to Improve Baseline Balance in Educational Experiments Kari Lock Morgan Department of Statistics Pennsylvania State University with Anna Saavedra and Amie Rapaport SREE March 1 st, 2018

Motivation • RCTs are the “gold standard” for estimating causal effects • WHY? – They eliminate confounding variables (balance covariates) – They yield unbiased estimates … on average! • For any particular experiment, covariate imbalance is possible (and likely!), and conditional bias exists

Typical RCT Randomize units to treatment groups Conduct experiment Check baseline balance Analyze results Why not check balance before conducting the experiment, when you can still fix it?

Rerandomization Collect covariate data Specify objective criteria for acceptable balance (Re)randomize units to Randomize units to treatment groups Check balance unacceptable Conduct experiment Analyze results

Context • Students learn Advanced Placement (AP) content through the Knowledge in Action (KIA) projectbased learning approach designed to develop students’ deeper learning of skills and content • RCT evaluation of KIA impact on student outcomes • Recruited teachers across five districts, teachers in 76 schools enrolled • Randomized at the school level within districts

KIA Covariates • Only previous cohort data available at the time of randomization • Covariates varied by district, but included – Standardized test scores (PSAT/AP/8 th grade) – Socio-economic status – % Nonwhite (some districts) – Course (APES or APGOV) (some districts)

KIA Criteria •

Covariate Balance: One District •

Covariate Balance

Outcome Precision •

More Power! • Significance for smaller effect sizes! • Use randomization test to take advantage of this; otherwise inference will be conservative

Regression •

Why Rerandomize? • Avoid bad/unlucky randomizations • Improve covariate balance • Increase power • Reduce reliance on assumptions

klm 47@psu. edu Morgan, K. L. , and Rubin, D. B. (2012). “Rerandomization to Improve Covariate Balance in Experiments, ” Annals of Statistics, 40(2): 1262 -1282. Morgan, K. L. and Rubin, D. B. (2015). “Rerandomization to Balance Tiers of Covariates, ” JASA, 110(512): 1412 – 1421.