Controls to Reduce Threats to Validity Graziano and

Controls to Reduce Threats to Validity Graziano and Raulin Research Methods: Chapter 9 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program. Copyright © Allyn & Bacon (2007)

Threats to Validity �Covered in Chapter 8 �Validity can be threatened in many ways ◦ Presence of confounding variables ◦ Unrepresentative samples ◦ Inappropriate statistical tests or violations of statistics assumptions ◦ Subject and experimenter effects �All these threats can be controlled Copyright © Allyn & Bacon (2007)

Control Procedures � General control procedures (applicable to virtually all research) � Control over subject and experimenter effects � Control through the selection and assignment of participants � Control through experimental design Copyright © Allyn & Bacon (2007)

General Control Procedures �Preparation of the setting ◦ Free of distractions that might interfere ◦ A natural setting increases external validity �Response Measurement ◦ Use reliable and valid measures �Replication ◦ Demonstrates that findings are consistent and robust Copyright © Allyn & Bacon (2007)

Types of Replication � Exact Replication ◦ Repeating a study using identical procedures to the original � Systematic Replication ◦ Using a theoretical or procedural change � Conceptual Replication ◦ Varying the operational definitions of the variables to get new research hypotheses Copyright © Allyn & Bacon (2007)

Subject and Experimenter Effects � Blind procedures ◦ Best control for expectancy effects ◦ Single-blind: The experimenter does not know what condition the participant is in ◦ Double-blind: Neither the experimenter nor the participant knows what condition the participant is in Copyright © Allyn & Bacon (2007)

Subject and Experimenter Effects �Automation ◦ Reduces contact between participants and the experimenter ◦ Gives the experimenter less opportunity to affect participants �Using objective measures ◦ Objective measure require less judgment ◦ Provides less opportunity for subtle experimenter biases to affect the data Copyright © Allyn & Bacon (2007)

Subject and Experimenter Effects �Multiple observers ◦ Reduces bias because it challenges observers to be as objective as possible ◦ Can measure amount of observer agreement (percent agreement or Kappa) �Using deception ◦ Hides purpose of the study from participants ◦ Balanced placebo design is a good example Copyright © Allyn & Bacon (2007)

Balanced Placebo Design � Separates the pharmacological effects from the expectancy effects of a medicine � A two-factor design ◦ Factor 1 is whether the person takes medication ◦ Factor 2 is whether the person thinks he or she is taking medication Copyright © Allyn & Bacon (2007)

Balanced Placebo Design People Led to Believe This design crosses the taking medication with the belief that medication is being used Taking Medication Actual Situation Not Taking Medication Copyright © Allyn & Bacon (2007) Not Taking Medication

Participant Selection � Can generalize only if your sample is representative � Populations and samples ◦ General population: all potential participants ◦ Target population: those participants you are interested in ◦ Accessible population: portion of target population that is available to the researcher ◦ Sample: drawn from the accessible population Copyright © Allyn & Bacon (2007)

Populations and Sampling This figure shows the relationship between the various populations ◦ General Population ◦ Target Population ◦ Accessible Population ◦ Sample Copyright © Allyn & Bacon (2007)

Sampling Procedures � Random sampling ◦ Every participant has an equal chance of being sampled � Stratified random sampling ◦ Random sampling within strata (subgroups) � Ad hoc samples ◦ Random sample from accessible population ◦ Must generalize cautiously ◦ Should describe sample to help define limits of generalization Copyright © Allyn & Bacon (2007)

Participant Assignment �Critical to internal validity �Assignment Procedures ◦ Free random assignment �Random assignment of participants to groups ◦ Matched random assignment �Random assignment of participants in matched sets to groups ◦ Other matching procedures �e. g. , match groups on key characteristics Copyright © Allyn & Bacon (2007)

Matched Random Assignment � Match on relevant variables ◦ Variables likely to affect the dependent measure ◦ Variables that show the largest variability in the population � Procedures ◦ Match in sets on the relevant variable �Set size is the number of groups in the study ◦ Randomly assign participants from the set, one to each group ◦ Keep track of matching data for the statistical analysis Copyright © Allyn & Bacon (2007)

Experimental Design � Main focus of Chapters 10 through 13 � Experimental design maximizes validity ◦ Need to also include the other control procedures covered in this chapter � Key elements of experiments ◦ One or more control groups ◦ Random assignment of participants to groups Copyright © Allyn & Bacon (2007)

Summary �Most threats to validity can be minimized with proper use of control procedures �Broad classes of control procedures ◦ General control procedures ◦ Control over subject and experimenter effects ◦ Control through participant selection and assignment ◦ Control through specific experimental design Copyright © Allyn & Bacon (2007)