Recent Work William R Shadish University of California

Recent Work William R. Shadish University of California, Merced

Recent and Current Research • Empirical Studies of Research Methods – Good nonrandomized experiments – Regression discontinuity designs • Analysis and Meta-Analysis of Single. Case Designs • Causal Theory

Nonrandomized Experiments • Empirical studies of the conditions under which nonrandomized experiments can approximate results from randomized designs. – Shadish, Clark & Steiner (2008) JASA – Cook, Shadish & Wong (2008) JPPAM

Results • Really good quasi-experimental design helps a lot – Especially careful attention to a well-matched intact control group. – Reduces the amount of bias to be reduced • Analytic adjustments (PS, ANCOVA) can work, but only if – You have good measures of the selection process – Those measures have high reliability

Regression Discontinuity Designs • Assignment to conditions based on a cutoff on a measured pretest variable • We have done similar empirical studies of the conditions under which RDD matches results from RE, with similarly encouraging results

Single-Case Designs • On the What Works Clearinghouse Technical Advisory Group, David Rindskopf and I (later joined by Larry Hedges) noted the absence of attention to results from single case designs (SCDs) in deciding what works. • The problem was lack of accepted statistical methods for the analysis and meta-analysis of those designs. • Our research – Developing a d-estimator for SCD – Continued development of multilevel models.

Causal Theory • Campbell’s theory of causal inference is over 50 years old (Campbell & Stanley, 1957) • Where does Donald Rubin and Rubin’s Causal Model fit in? – Forthcoming special section of Psychological Methods on this. • Where does Judea Pearl’s graphical approach to causal inference fit in? – Current area of study – In preparation chapter in an APA Handbook of Research Methods probably to come out in 2011 or 2012.

Anticipating Some Questions • Are we revising Shadish Cook and Campbell? – Not yet, but we are actively considering it. – It would be nice first to solve the problem of the analysis of SCD (as it relates to the analysis of short time series) • Are we revising Shadish Cook and Leviton (Foundations of Program Evaluation)? – Sadly, probably not. – I’m getting a little out of touch with evaluation theory, as a lot has happened in the last 20 years.

Further Readings • • • Cook, T. D. , Shadish, W. R. , & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724750. Shadish, W. R. , Rindskopf, D. M. & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 3, 188 -196. Shadish, W. R. , Clark, M. H. , & Steiner, P. M. (2008). Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment. Journal of the American Statistical Association, 103, 1334 -1343. Shadish, W. R. , & Cook, T. D. (2009). The Renaissance of Field Experimentation in Evaluating Interventions. Annual Review of Psychology, 607 -629. Shadish, W. R. (in press). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods.

THANKS! To My Many Coauthors (Tom Cook, MH Clark, Peter Steiner, Larry Hedges, David Rindskopf, Vivian Wong) and To You