A Comparison of Propensity Score Stratification Methods M

A Comparison of Propensity Score Stratification Methods M. H. Clark, Ph. D. University of Central Florida American Evaluation Association Convention October 15, 2014

Introduction v Propensity Score Adjustments is a popular method for reducing bias in non-randomized experiments by balancing treatment conditions on observed covariates. v These adjustments are made using a variety of methods: • • Matching Weighting Covariate adjustment Stratifying

Introduction v Stratifying on propensity scores divides cases into intervals so each strata has a group of treated and control participants. • Treated and control units should be comparable within each strata v Strata are often determined by • Proportion of total cases (20% in each strata) • Range cut-offs (0 -. 2, . 201 -. 4, ect. ) v Strata have unequal sample sizes across groups (i. e. , nt = 5, nc = 25 in strata 1)

Introduction v Two common methods for stratification: • Between-subjects analysis (e. g. , t-test) for each stratum and average effect sizes across strata [1, 2] • Treatment-by-strata factorial ANOVA to estimate the main effect for treatment [3, 4] v Common support can be improved by providing more control cases [5] v Varying strata affects bias and support [6] • Five strata can reduce 90% of bias [7]

Introduction v Hypotheses • Bias and covariate balance will differ depending on stratification method • Bias will decrease and covariate balance will improve as strata and control cases increase v Design • Two stratification methods: average of strata and strata x treatment ANOVA • Number of strata: 3, 5, and 10 • Sample size ratio: 1 to 1, 1 to 2, 1 to 3 • Bias, covariate balance, common support

Data Source v Original data N=611 • 236 were randomly assigned to math or vocabulary training • 103 self-selected into math • 268 self-selected into vocabulary v True treatment effects were removed v Cases were removed to get: • n = 80 in treatment (math) • n = 240 in comparison (vocabulary)

Results: Bias Reduction

Results: Bias Reduction

Results: Common Support Strata n. T 1 n. C Strata n. T 1: 1 1: 2 1: 3 5 39 66 102 1 2 20 19 60 86 3 55 22 34 52 Strata n. T n. C 1: 1 1: 2 1: 3 1 1 19 37 63 2 6 25 47 58 3 11 10 33 53 4 28 14 23 36 5 34 12 20 30 n. C 1: 1 1: 2 1: 3 1 9 16 31 2 0 10 21 32 3 3 16 23 29 4 3 9 24 29 5 5 4 19 27 6 6 6 14 26 7 14 6 11 18 8 14 8 12 18 9 15 6 12 17 10 19 6 8 13

Results: Covariate Balance with a sample size ratio of 1 to 1 (nt = 80, nc = 80) Covariate Unadjusted Average of 10 Strata ANOVA with 5 Strata 9. 31 -57. 18 23. 90 65. 44 22. 77 5. 03 Literature Experience -21. 42 1. 22 2. 66 Math Anxiety -20. 14 12. 62 6. 30 Prefer Math 56. 952 3. 81 7. 88 Liked Intervention Topic 36. 11 -12. 65 95. 21 Avoid Intervention Topic -113. 41 9. 15 0. 00 Improve in Intervention Topic 26. 85 -3. 01 -129. 40 Extraversion 94. 42 3. 16 5. 80 -45. 49 -34. 00 -48. 24 Math Pretest Math Experience White Positive SB indicates higher scores for or more in the treatment group

Results: Covariate Balance with a sample size ratio of 1 to 3 (nt = 80, nc = 240) Covariate Unadjusted Average of 10 Strata ANOVA with 5 Strata Math Pretest 22. 29 -20. 82 11. 89 Math Experience 72. 77 4. 97 2. 59 Literature Experience -36. 11 -15. 72 8. 78 Math Anxiety -26. 43 4. 53 1. 74 Prefer Math 56. 95 10. 71 26. 46 Liked Intervention Topic 36. 11 5. 50 82. 92 Avoid Intervention Topic -113. 41 5. 47 0. 00 26. 85 10. 66 -63. 02 Extraversion -54. 13 -4. 89 1. 00 White -45. 49 -13. 60 7. 50 Improve in Intervention Topic Positive SB indicates higher scores for (or more) in the treatment group

Conclusions v Biased was reduced with • Factorial ANOVA • Five strata v Covariate balance wasn’t consistent across covariates or methods • Improved with more comparison cases v Common Support • Improved with fewer strata

References 1. 2. 3. 4. 5. 6. 7. Austin & Mamdani (2006) Shadish, Clark & Steiner (2008) Clark (in press) Maxwell & Delany (2004) Bia (in press) Luellen, Shadish & Clark (2005) Cochran (1968)