Design 3 quasiexperimental and nonexperimental designs Learning outcomes

Design (3): quasi-experimental and non-experimental designs • • Learning outcomes State characteristics of quasiexperimental design State and define different types of quasiexperimental design and their analysis State characteristics of non-experimental design State and define different types of nonexperimental design and their analysis RDA 2 (PSY 2001 -N) 1

Outline • Quasi-experimental designs – The meaning of treatment – Some elementary quasi-experimental designs • Non-experimental designs – – – Predictive versus explanatory research Categorical IVs Continuous IVs Categorical and continuous IVs Longitudinal research RDA 2 (PSY 2001 -N) 2

Experiment • Defining characteristics: – Establish cause-effect relationship between IV(s) and DV as stated in experimental hypothesis – Manipulation of IV(s) – Random assignment of participants to levels (categories) of the IV(s) • If no random assignment then quasiexperiment • If neither manipulation nor random assignment then non-experiment RDA 2 (PSY 2001 -N) 3

Quasi-experimental designs • Quasi-experiment: as an experiment, except that participants are not randomly assigned to groups • Difficulties: – How to identify and separate the effects of treatments from the effects of all other factors affecting the DV? – Possible selection-treatment interaction: the groups may differ on a range of variables that may affect the effect of treatment on the DV • Meaning of treatment: what is it about the treatment that presumably caused group difference on the DV? • Greater clarity is possible when researcher is involved in design and administration of treatment • Poor examples of ‘treatment’, involving ‘global settings’ as treatment: compensatory education programmes (e. g. Head Start), institutions (public versus state schools) • Non-experimental design: no treatment administered RDA 2 (PSY 2001 -N) 4

Two quasi-experimental designs • Non-equivalent control group • Interrupted time series RDA 2 (PSY 2001 -N) 5

Non-equivalent control group • Difference with ‘Treatment-control. Pre-measures and post-measures’ design: no randomization possible selection bias other threats to internal validity even more likely, e. g. maturation • Design without a pre-test is preferable (as in experimental designs) • The further apart treatment and control group on pre-test, the more likely are effects of • selection and • Interaction of selection with other factors RDA 2 (PSY 2001 -N) 6

Non-equivalent control group (2) • Analysis: • No single preferred analysis • Some recommend multiple analyses • Others stress importance of correct model specification as a basis for analysis, i. e. correct identification of IV(s) and extraneous variable(s), and relationships among them and with the DV • Two main approaches to analysis 1. Regression adjustment 2. Difference scores RDA 2 (PSY 2001 -N) 7

Regression adjustment • Adjust post-test score for initial differences between NE groups on pre-test based on regression analysis • Mathematically equivalent to analysis of co-variance • However, threats to validity 1. Measurement issue: factor structures in the two comparison groups may not be equal and may change over time • pre-test and post-test should measure the same construct in both groups, and pre-test and post-test should measure the same construct 2. Regression artefacts • The further the two groups apart on the pre-test the greater the threat of regression towards the mean RDA 2 (PSY 2001 -N) 8

Analysis of difference scores • Procedure • Per participant subtract pre-test score from post-test score – ‘raw-score’ difference • Calculate mean difference for both groups • Test difference between mean of difference for significance, e. g. using t test • Difference scores = specific case of regression adjustment • Same threats as those associated with RA • Additional threats follow RDA 2 (PSY 2001 -N) 9

Analysis of difference scores - threats • Difficulties in interpretation and analysis • Participants in experimental and control groups may ‘grow’ at different rates. The larger the difference on the pre-test the more difficult to interpret the results. • Sensitivity and difficulty level of pre-test • When ceiling and/or floor effects occur on pre-test or post-test difference scores are not meaningful • ‘Correlation with initial status’ • Would expect a positive (imperfect) correlation between pre-test and post-test • However, with equal SD of pre-test and post-test, correlation will be negative those scoring high on the pre-test tend to have smaller gain scores (or even decreasing scores) and those scoring low on pre-test tend to have larger gain scores RDA 2 (PSY 2001 -N) 10

Interrupted time series (1) • Aims of collecting times-series data • Develop models to explain patterns that occur over time • Use models forecasting • Interrupted time series • Series interrupted by some discrete event or intervention • Purpose: assess the effect of intervention; in what way has the intervention changed the time-series data? • Costly • Attrition rates can be high RDA 2 (PSY 2001 -N) 11

Interrupted time series (2) Simple interrupted time series • Major threat: history Interrupted time series with non-equivalent control group Analysis: time series modelling/analysis RDA 2 (PSY 2001 -N) • Attempts to control for history and other threats to internal validity • Degree to which threats to internal validity are controlled depends on comparability of the two NE groups 12

Summary – quasi-experimental designs • In quasi-experimental designs participants are not randomly assigned to groups • Difficult to identify and separate the effects of treatments from the effects of all other factors affecting the DV • Possible selection-treatment interaction • Two elementary quasi-experimental designs: • Non-equivalent control group • Interrupted time series RDA 2 (PSY 2001 -N) 13

Predictive versus explanatory research • Predictive research – – Aim: develop systems to predict criteria Predictor(s) and criterion Choice of predictor set does not require theory Predictors selected until best sub-set is found • Explanatory research – – – Aim: test hypotheses to explain phenomena of interest Independent variable's) – IV(s) (presumed cause) and dependent variable – DV (presumed effect) Choice of predictor set based on theory • In the context of non-experimental designs, we will only consider explanatory research RDA 2 (PSY 2001 -N) 14

Formulation and testing of models • Direction of inference: – – • • From IVs (cause) to DVs (effect) in (quasi)-experimental research From DVs to IVs in non-experimental research; chance of confusing IV and DV is far greater in non-experimental research In (quasi)-experimental research, groups are compared that are exposed to different treatments In non-experimental research groups are very frequently formed on the basis of the DV and differences among groups are attributed to some cause Researcher tries to find out what variable explains observed differences However, groups that are compared may not have been exposed to different treatments RDA 2 (PSY 2001 -N) 15

Threats to internal validity in non-experimental designs • Use of non-probability samples: – – Failure to use probability samples (random selection of participants from population and each has a non-zero probability of being selected) This is not a threat to internal validity in experimental designs • Uncontrolled confounding variables – Control by • Subject selection (see above) • Statistical adjustment (same problems as with quasi-experimental design) – – Exercise of controls in non-experimental research may have adverse effects by distorting relations among variables Some elementary non-experimental designs follow RDA 2 (PSY 2001 -N) 16

Categorical IV • IV: one or more broad classifications of people are used to explain the status of participants on some phenomenon of interest (presumed DV) • Probability sampling is a necessary, but not sufficient condition for valid inferences • Each grouping or class is treated as a separate population for comparison • Difficulty in offering IV as an explanation for observed differences on DV – ‘pseudo explanations’: ‘a sex difference is a question, not an answer’ (Baumeister) RDA 2 (PSY 2001 -N) 17

Categorical IV (2) • One IV – Analysis: unrelated t test or one-way independent measures analysis of variance • Multiple IVs: similarity with factorial designs in experimental research is only superficial – Influential extraneous variables may have been overlooked – The IVs are (usually) not independent of each other, as they are in experimental research RDA 2 (PSY 2001 -N) 18

One continuous IV • Omission of relevant variables correlated with IV is a specification error and results in biased estimates of the effects of the IV • This is unavoidable in non-experimental research; minimize this shortcoming BY including major relevant variables in the design • Therefore non-experimental designs with one IV are almost certainly flawed • Analysis: simple regression analysis RDA 2 (PSY 2001 -N) 19

Multiple continuous IVs • Correct model specification depends on theory of the phenomenon under investigation regarding the nature of the relations among the IVs RDA 2 (PSY 2001 -N) 20

Multiple continuous IVs (2) – Single-stage models SES MA MOT AA error • DV affected by a set of intercorrelated IVs • Exogenous variable: variability assumed to be determined by causes outside the model • Endogenous variable: variability explained by exogeneous variables and possibly other endogenous variables – Analysis: (standard) 21 multiple regression analysis RDA 2 (PSY 2001 -N)

Multiple continuous IVs (3) SES error MA error AA MOT SES MOT error MA AA • Multi-stage models – One or more exogenous variables and two or more endogenous variables – Stages: the number of endogenous variables • Analysis: path analysis; also hierarchical multiple regression analysis RDA 2 (PSY 2001 -N) 22

Categorical and continuous IVs • E. g. IVs: years of experience and gender; DV: salary • Years of experience as a control variable, investigating the effect of gender after adjusting for years of experience OR • Years of experience as a moderator of gender, investigating an attribute-treatment interaction • Analysis technique: analysis of co-variance, but interpretation different from that in experimental and non-experimental designs RDA 2 (PSY 2001 -N) 23

Longitudinal research • Longitudinal: study a phenomenon repeatedly as it exists and evolves over time • Cross-sectional: measurement at a single moment in time • Disadvantages of longitudinal research: • • • Costly Attrition Effects of repeated measurement (testing) The meaning of measures may change across time Changes in personnel History Information diffusion Cannot provide answers to pressing questions Interest in research question and theoretical framework may have changed by the time an answer becomes available RDA 2 (PSY 2001 -N) 24

Preparation for next practical class • Study key experimental research designs • Reading: Pedhazur: Ch. 13, 14 • Clark-Carter: Ch. 1 • Lecture notes RDA 2 (PSY 2001 -N) 25

Summary – non-experimental designs • Non-experimental designs: • Participants are not randomly assigned to groups (a) difficult to identify and separate the effects of treatments from the effects of all other factors and (b) possible selection-treatment interaction (as in QE designs) • No treatment is administered difficult to identify and isolate the effect of IV(s) • Threats to internal validity include • Use of non-probability samples • Uncontrolled confounding variables • One or more categorical or continuous IVs or a combination of categorical and continuous IVs 26 can be used RDA 2 (PSY 2001 -N)