Foundations of Research 10 Basic research designs 1

Foundations of Research 10. Basic research designs. 1 This is a Power. Point Show ü Click “slide show” to start it. ü Click through it by pressing any key. ü Focus & think about each point; do not just passively click. ü To print: ü Click “File” then “Print…”. ü Under “print what” click “handouts (6 slides per page)”. © Dr. David J. Mc. Kirnan, 2014 The University of Illinois Chicago Mc. Kirnan. UIC@gmail. com Do not use or reproduce without permission

Foundations of Research n n Basic experimental designs This module overviews the core elements of an experimental research design. We will discuss “pre-experimental” designs Ø These typically have no control group or may use existing groups Ø They are often used in preliminary or exploratory research n “True” experiments have several key characteristics: Ø A control group Ø Random assignment of participants to groups Ø Standardized or uniform procedures for each group 2

Foundations of Research 3 Experimental designs and validity n We will discuss internal and external validity. n Internal validity Ø In experiments we manipulate (induce…) the Independent Variable. Ø We then measure the Dependent Variable. Ø Experimental hypothesis: the outcome (the level of the Dependent Variable) is caused by – and only by – the Independent Variable. Ø Internal validity: How confident are we that the outcome was due only to the Independent Variable. Ø Confound: A variable other than the IV that caused or influenced the result. Ø Did the participants in the experimental v. control groups differ on something other than the IV? Ø Were the procedures biased in some way…? Confound

Foundations of Research n Experimental designs and validity 4 External validity Ø Experimental participants are a sample of the larger population. Ø The experimental manipulation attempts to accurately induce the Independent Variable. Ø The outcome measure represents the Dependent Variable. Ø The experiment is conducted in a specific physical or cultural setting. Ø External validity: Ø Does the research sample accurately represent the larger population? Ø Does the experimental manipulation accurately represent the concept we think causes the outcome or results? Ø Do the outcome measures accurately represent the phenomenon we are trying to explain? Ø Is the experimental setting representative of how these processes work in nature?

Foundations of Research External validity: summary 5 Is the sample representative of the larger population? Does the outcome measure represent what we are trying to explain? The research Sample: The Dependent Variable The study The research structure Setting: & context The Independent Variable Does the experimental manipulation actually create the phenomenon you are interested in? Is this typical of the natural settings where the phenomenon occurs?

Foundations of Research n Overview: Basic Designs 6 “Pre-experimental” designs: no control group Post-Test Only Design Pre- Post- Test Design Group assignment Pre-test Experimental manipulation Outcome Experimental Observe 1 Treatment Observe 2

Foundations of Research n Basic Designs 7 “Pre-experimental” designs: no control group Post-Test Only Design Pre- Post- Test Design n True (or Quasi-)experimental designs with a control group “After only” Control group design Pre- Post- Group Comparisons Group assignment Pre-test Experimental manipulation Outcome Experimental Observe 1 Treatment Observe 2 Control Observe 1 Control Observe 2

Foundations of Research n Basic Designs 8 “Pre-experimental” designs: no control group Post-Test Only Design Pre- Post- Test Design n True (or Quasi-)experimental designs with a control group “After only” Control group design Pre- Post- Group Comparisons Multiple group comparison Group assignment Pre-test Experimental manipulation Outcome Experimental Observe 1 Treatment 1 Observe 2 Experimental Observe 1 Treatment 2 Observe 2 Control Observe 1 Control Observe 2

“Pre-experimental” designs Foundations of Research Post-Test Only Design Group Only 1 “group”. ü A single set of physical measures or observations ü In behavioral science typically an existing group: no selection or assignment occurs. Treatment Measure The condition or experimental intervention (“Treatment”) may or may not be controlled by the researcher. Measurement may or may not be controlled by the researcher. ü In Earth Sciences we may examine how a geologic formation is associated with historical water flow. ü E. g. ; existing (archival) climate data. ü In Behavioral Sciences we may examine naturally occurring or system-wide events • e. g. , socio-economic conditions and racial conflict, • the effect of a government policy change on foreclosure rates…. ü A survey after an event such as 9/11 ü Uniform crime rates, hospital admissions, etc. 9

Foundations of Research “Pre-experimental” designs 10 Post-Test Only Design Group Treatment Measure Pre- Post- Test Design Group Only one group; • only group available? • naturally occurring intervention? Measure 1 Treatment Measurements from a baseline period and after an intervention or naturally occurring event. Measure 1 All participants get the same treatment, which may or may not be controlled by the researcher. ü Comparing archival climate data from before & after industrialization ü Examining school test scores before & after the introduction of the STEM educational approach

Foundations of Research “Pre-experimental” Designs (2) Advantage of “Post-” & “Pre- Post-” Designs: Allow us to study naturally occurring interventions. ü e. g. , test scores before and after some school change, ü Crime rates after a policy change, etc. ü Having both Pre- and Post measures allows us to examine change. 11

Foundations of Research “Pre-experimental” Designs (2) Disadvantage of “Post-” & “Pre- Post-” Designs: No control group = many threats to internal validity. n n n Maturation: Participants may be older / wiser by the post-test History; Cultural, historical or physical events may occur between pre- and post-test that can represent a confound in our analysis Mortality: Participants may non-randomly drop out of the study Regression to baseline: Participants who are more extreme at baseline look less extreme over time as a statistical confound. Reactive Measurement: Scores may change simply due to being measured twice, not the experimental manipulation. 12

Foundations of Research Experiments 13 “After only” Control group design Experimental Control Treatment 2 Observe 2 Control Observe 2 Adds a control group. Either… Observed Groups: Measure Dependent Variable(s) only at follow-up. ü ü Use experimental or standard measures (e. g. , grades, census data, crime reports). Naturally occurring (e. g. , Class 1. v. Class 2) or Self-selected (sought therapy v. did not…). Assigned Groups: ü Randomly assign participants to experimental v. control group, or ü Match participants to create equivalent groups.

Foundations of Research 14 Advantages of experimental design “After only” Control group design Experimental Control Advantage: Treatment 2 Observe 2 Control Observe 2 Lessens the likelihood of confounds or threats to internal validity. ØControl group ØRandom assignment Disadvantage: Existing or self-selected groups may have confounds. No baseline or pre- measure available: Ø We cannot assess change over time. Ø We cannot assess whether the groups are equivalent at baseline.

Foundations of Research Basic Designs: True experiments Pre- Post- Group Comparisons Group 1 Measure 1 Group 2 Measure 1 (most common study design) Two groups: Observed (quasi-experiment) or Assigned (true experiment). “Groups” can be ü Different physical conditions or lab preparations, ü Existing blocks of people ü Actual experimental groups… Baseline (“pre-test”) measure of study variables and possible confounds. 15

Foundations of Research Basic Designs: 16 True experiments (2) Pre- Post- Group Comparisons (most common study design) Group 1 Measure 1 Treatment Measure 2 Group 2 Measure 1 Control Measure 2 The group getting the experimental condition is contrasted with a control group. . ü Naturally occurring ü Created by experimenter “Post-test” follow-up of dependent variable(s); ü Simple outcome ü Change from baseline.

Foundations of Research Basic Designs: 17 True experiments (3) Pre- Post- Group Comparisons (most common study design) Group 1 Measure 1 Treatment Measure 2 Group 2 Measure 1 Control Measure 2 Advantages: Pre-measure assesses baseline level of Dependent Variable Ø Allows researcher to assess change Ø Can find matched pairs of participants or physical samples and assign each to different groups (rather than random assignment). Ø Can assess whether groups are equivalent at baseline. Disadvantage: Highly susceptible to confounds if using observed or self-selected groups.

Foundations of Research More Complex Experimental Designs Multiple group comparison Group 1 Measure 1 Treatment #1 Group 2 Measure 1 Treatment #2 Group 3 Measure 1 Control ü 3 (or more) groups ü Typically formed by Random assignment. Multiple experimental groups, e. g. PLow drug dose, PHigh drug dose, PPlacebo. or PMale therapist, PFemale therapist, PWait list control. 18

Foundations of Research 19 More Complex Experimental Designs Multiple group comparison Group 1 Measure 1 Treatment #1 Measure 2 Group 2 Measure 1 Treatment #2 Measure 2 Group 3 Measure 1 Control Measure 2 Compare: ü Experimental group 1 from experimental group 2. ü Either / both experimental groups from the control group.

Foundations of Research 20 More Complex Experimental Designs Multiple group comparison Group 1 Measure 1 Treatment #1 Measure 2 Group 2 Measure 1 Treatment #2 Measure 2 Group 3 Measure 1 Control Measure 2 Advantages: Test dose or context effects: ü Drug doses, amounts of psychotherapy, levels of anxiety, etc. Increasing dose effect can be tested against no dose. ü Diverse conditions to test 2 nd hypotheses or confounds, e. g. , therapy delivered by same sex v. opposite sex therapist. Disadvantage: ü More costly and complex. ü Potential ethical problem with a “no dose” (or very high dose) condition.

Foundations of Research Core components of a research study 21 We will use this framework to think about the basic elements of an experiment. Participant Selection Who or what are we studying? How did we recruit or sample them? Participant Assignment Experimental Procedures Experimental Treatment or Manipulation We will have at least one Experimental Group and a Control Group. What Experimental & instructions do control we give? groups get different What conditions. experimental How do we assign participants or samples to be in one or the other? tasks will We hypothesize participants be that this performing? manipulation “causes” the What measures outcome. might we be taking? Results What outcomes are we measuring? What is the experiment trying to explain?

Foundations of Research Participant Selection Sample We recruit a sample of participants from the larger population. Experimental design overview Participant Assignment Experimental Procedures 22 Experimental Treatment or Manipulation Results Group A Procedure Treatment Outcome Group B Procedure Control Outcome (Group C) (Procedure ) (Alternate Treatment? ) We randomly assign them to groups to ensure the groups are equivalent at baseline. Procedures for all groups should be exactly the same… (Outcome) …except the experimental manipulation, i. e. , the Independent variable. Hypothesis: The outcome or Dependent Variable varies only by group.

Overview of true experimental designs Foundations of Research Participant Selection Sample Participant Assignment Experimental Procedures Experimental Treatment or Manipulation 23 Results Group A Procedure Treatment Outcome Group B Procedure Control Outcome (Group C) (Procedure ) (Alternate Treatment? ) Experimental group Control group (Outcome)

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample Does the sample well represent the population? External validity Random selection Experimental Treatment or Manipulation 24 Results Group A Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) • Was recruitment biased? • Is the sample size large enough? What form of validity is threatened by sample bias? What can we do to avoid that threat?

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample Does the sample well represent the population? Results Group A Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) Are the groups equal at baseline? External Internal Random selection Random Assignment validity Experimental Treatment or Manipulation 25 validity • Did participants Self-select (in or out) of the study? • Did we use existing groups? Validity Threat? Solution?

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample Does the sample well represent the population? Random selection Results Group A Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) Procedures Are the groups equal the same for all groups? at baseline? External. Validity Threat? Internal validity Experimental Treatment or Manipulation 26 Solution? validity Random Assignment Internal validity: Lack of confounds • Do both groups have the same expectations? • Are participants (and researchers) really blind? • Do we treat both groups the same?

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample validity Random selection Results Group A Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) • Does the operational definition Procedures Does the Are the same for really express the construct we are sample well groups equal all groups? interested in? represent the at baseline? • Have we given the correct dose of population? the IV? External Experimental Treatment or Manipulation 27 Internal. Validity Threat? validity Random Assignment validity: Solution? Lack of confounds Independent variable faithfully reflects the construct? External Validity Correct IV?

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample Does the sample well represent the population? External validity Random selection Experimental Treatment or Manipulation 28 Results Group A Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) Procedures Independent Are the same for • Is any difference we see variable actually groups equal all groups? statistically significant (reliable faithfully& at baseline? reflects the meaningful)? construct? • …or it is due to chance alone. . Internal validity Random Assignment Internal External validity: Validity Threat? Validity Solution? Lack of Correct IV? confounds Groups really different at outcome? Internal Validity: Statistical testing

Foundations of Research Overview: experimental designs Participant Experimental Recruitment Assignment Procedures Sample Procedure A Treatment Outcome Group B Procedure A Control Outcome Group C Alternate Outcome Procedure A Treatment (? ) Procedures the same for all groups? External Internal Random selection Random Assignment validity Results Group A Are the groups equal at baseline? Does the sample well represent the population? Experimental Treatment or Manipulation 29 validity: Lack of confounds Independent Groups really variable different at faithfully outcome? reflects the construct? External Internal Correct IV? Statistical testing Validity:

Foundations of Research Why are research methods so important? A case study. Do workplace health programs actually save money? 30 Baxter, S. et al. , (2014). The Relationship Between Return on Investment and Quality of Study Methodology in Workplace Health Promotion Programs. American Journal of Health Promotion, Vol. 28 (6), Pp. 347 -363. Click for abstract. § Over the past 20+ years there has been considerable interest in workplace health promotion: Ø …dietary, “lifestyle” and exercise advice & resources; Ø …smoking cessation, weight loss programs…

Foundations of Research Why are research methods so important? A case study. 31 Do workplace health programs actually save money? § The hypothesis is that healthier employees will save the company money, via lower absenteeism, health insurance costs, etc. § Evidence appears to support that claim, or does it? § Slyan et al. took published studies and divided them into four categories: § Randomized Controlled Trials; “true experiments”, the gold standard of research. § Quasi-experimental designs; where participants were able to choose whether to get the health program or not (self-selection into groups). * § Non-experiments; basically anecdotal or observational studies. § Modeling studies; predicting outcomes based on extant data on the general effects of healthier behavior. * We will discuss quasiexperiments next module.

Foundations of Research 32 Why are research methods so important? A case study. Do workplace health programs actually save money? § The results showed clearly that “Return On Investment” (ROI; actual savings) was higher as methodological quality went down. High quality studies showed a very modest ROI Whereas low quality studies showed substantial ROI In low quality studies, companies appeared save more than twice the money they invested in health promotion.

Foundations of Research 33 Why are research methods so important? A case study. Do workplace health programs actually save money? § Comparing Randomly Controlled Trials (RCTs) to others was particularly damning for the hypothesis…. RCTs showed companies to actually lose money through health promotion. Non-experimental and modeling studies showed significant ROI Lower-quality research lead to very misleading results

Foundations of Research Why are research methods so important? A case study. 34 Do workplace health programs actually save money? § Why this huge difference between randomized controlled trials and non-experiments? ü In the non-randomized trials employees were able to choose (selfselect) which group they wanted to be in. ü It is completely plausible that healthier or more motivated employees would join the health group, not the control group. ü Studies with self-selection may be simply showing us that healthier people stay healthy and cost less, not that the actual programs did anything.

Foundations of Research Experimental design key elements n Control group v. non-controlled designs n Threats to internal validity: SUMMARY n Overview: key terms § § § Maturation History Mortality Regression to baseline Reactive Measurement n “Pre-experimental” designs n Pre-post designs n Multiple group comparisons 35

SUMMARY Foundations of Research Overview: experimental designs Participant Recruitment Participant Assignment Does the sample well represent the population? Are the groups equal at baseline? External validity Internal validity Experimental Procedures the same for all groups? Internal validity 36 Exp. Treatment or Manipulation Results Independent variable faithfully reflects the construct? Groups really different at outcome? External validity Internal validity

Foundations of Research Please go on to the Research Designs quiz. 37