QuasiExperimental Designs Manipulated Treatment Variable but Groups Not

Quasi-Experimental Designs Manipulated Treatment Variable but Groups Not Equated

Pretest-Posttest Nonequivalent Groups Design N O X O N O O • Cannot assume that the populations are equivalent prior to treatment. • Selection and Selection Interactions are threats to internal validity. • Can try to select subjects or intact groups in ways that make it likely that the groups are similar, but what about unknown variables on which the groups may differ.

Double-Pretest Nonequivalent Groups Design N N O O X O O • Some control for Selection x Maturation. • If groups are maturing at different rates, that may be shown in the two pretests.

Regression-Discontinuity Design C C O O X O O • ‘C’ indicates subjects are assigned to groups based on score on covariate. • Groups are deliberately nonequivalent. • I shall illustrate with a hypothetical example

Evaluating an Online Tutorial • IV = Student completes online tutorial or not. • DV = Student’s score on statistics course exam. • Pretest/Covariate = Student’s score on a test of statistics aptitude. • How do I assign students to groups?

Evaluating an Online Tutorial • • Let the students self-select into groups. This gives me a pretest-posttest nonequivalent groups design. I use pretest scores as covariate in ANCOV. This does not, however, remove all possible confounds. How might the groups have differed other than on statistics aptitude? ? ?

Evaluating an Online Tutorial Randomly assign students to groups. • This would be an experimentally sound, randomized pretest-posttest control group design. • And you would live to regret trying it. – Students complain. – Their parents complain. – The Chair of the Department intervenes. – The IRB revokes your authority to do research.

Evaluating an Online Tutorial Try a switching-replications design. • Those who have to wait until the second half of the class would be disadvantaged – if you don’t learn the beginning material well, the later material will very hard to learn. • Those who have it taken away at midsemester will complain.

Evaluating an Online Tutorial Apply the treatment only to those most in need of it, those lowest in statistics aptitude. • Those not selected may complain that they could benefit from it too. – Tough, there is an American tradition of favoring the underdog.

Evaluating an Online Tutorial • Those selected may complain about having to do extra work. – You can’t please everybody every time. Convince them they need to do it. • May be cases when you want to give the tutorial only to those with highest aptitude – purpose of tutorial is to allow brightest students to finish course early, allowing prof more time to spend in class with others.

Evaluating an Online Tutorial: Regression Discontinuity • Suppose I pick a cutoff on the covariate (stats aptitude) so that the lower ½ get the treatment, upper ½ don’t. • Simulated data are in file Reg. D 0. txt. • C = control group, T = Treatment group. • 2 nd score is posttest score. • 3 rd score is pretest score. • I defined the treatment effect to be zero in the population.

Evaluating an Online Tutorial: Population Treatment = 0 • In the population, Post = 7 + 1. 35 Pre + error, =. 9, • In the sample, ignoring group, Post = 7. 58 + 1. 27 Pre + error, r =. 85, and MSE = 2. 13 • Look at this plot of the data:

Ignoring Group

Evaluating an Online Tutorial • Now I compute two separate regressions, one for each group. • T: Post = 8. 09 + 1. 17 Pre + error, r =. 62, and MSE = 2. 13. • C: Post = 6. 33 + 1. 43 Pre + error, r =. 72, and MSE = 2. 29. • The plot shows how little the two lines differ:

Separate Regressions, by Group, No Treatment Effect in Population

Evaluating an Online Tutorial • I re-simulated the data, with a 3 point treatment effect built in. • The data are at Reg. D 1. txt. • T: Post = 11. 27 + 1. 07 Pre + error, r =. 82, and MSE = 1. 35. • C: Post = 7. 90 + 1. 18 Pre + error, r =. 82, and MSE = 1. 25 • The plot shows a clear regression discontinuity:

Separate Regressions, by Group, 3 Point Treatment Effect

Evaluating an Online Tutorial • The dotted line shows the expected regression for the treatment group if there were no treatment effect. • Hard to imagine how any threat to internal validity would create the observed regression discontinuity. • Caution: This analysis assumes the regression is linear, not curvilinear.

Proxy-Pretest Design • • N O 1 X O 2 N O 1 O 2 You have a nonequivalent groups posttest only control group design. The treatment has already been administered. Now you decide you want a pretest too. Can’t warp time, can find an archival proxy pretest.

PSYC 2210 and Understanding Stats • Mid-semester, I ask myself “does taking 2210 improve students understanding of stats? ” • I’ll compare students in current 2210 class with those in another class (excluding any who have already taken 2210). • I want a pretest too, but the treatment is already in progress.

PSYC 2210 and Understanding Stats • I use, as a proxy pretest, students’ final averages from PSYC 2101. • Conduct an ANCOV – IV = took 2210 or not – DV = end of course stats achievement test – COV = the proxy pretest

Separate Pre-Post Samples Design • Pretest subjects different than posttest subjects. • I want to evaluate online tutorial in stats. • Both I and my friend Alex taught stats this semester and last semester. • Both of us gave our students a end of course standardized exam. N N O X O O O

Separate Pre-Post Samples Design • Row 1: My students last semester, no tutorial. • Row 2: My students this semester, with tutorial. • Row 3: Alex’s students last semester, no tutorial • Row 4: Alex’s students this semester, no tutorial. • Selection problems likely. N O N N N X O O O

Nonequivalent Groups Switching Replications Design N N O O X O O • I am teaching two sections of stats. • I make the experimental tutorial available the first half semester to one class • and the second half semester to the other. • Might reduce complaints, until students from the two classes meet each other.

Nonequivalent Dependent Variables Design • • • Only one group of subjects, but two DVs. One DV you expect to be affected by X. The other you expect not to be affected by X. The second DV serves as a control variable. Should be similar enough to 1 st DV that it will be affected in same way by history, maturation, etc.

Nonequivalent Dependent Variables Design • I want to evaluate effect of stats remedial tutorial given to all PSYC 2210 students. • DV 1 = stats knowledge measured at start and end of semester. • DV 2 = Vocabulary test given at start and end of semester. • More impressive if have multiple control variables and an a priori prediction of extent to which each will be affected.

Nonequivalent Dependent Variables Design • • • Stats Knowledge (DV 1) – most affected Logical Thinking – next most Verbal Reasoning – same as LT Arithmetic Skills – next most Vocabulary – next most Artistic Expression – least affected by treatment

Regression Point Displacement Design N(n N • • = 1) O O X O O Only one subject in the treatment group Several or many in the control group. X = novel economic development plan. Treatment unit = your hometown, in which the plan was just initiated.

Regression Point Displacement Design • You consult state economic database. • Pick 20 cities comparable to your city, these serve as the control group. • Pretest = value of criterion variable (such as unemployment rate) last year. • Posttest = value of same variable two years later.

Regression Point Displacement Design • Plot Post x Pre for the Control Group. • Draw in regression for predicting Post from Pre. • Plot experimental unit data point. • If it is displaced well away from regression line, you have evidence of a treatment effect.