Experimental Research Design Causality Validity Threats to Validity

Experimental Research Design Causality & Validity • Threats to Validity – Construct (particular to experiments) – Internal – External – already discussed

Threats to Construct Validity • Mono Method bias – Remove common method variance – Establish convergent & discriminant validity • Hypothesis guessing • Inappropriate level of construct – Usually lower than what occurs in natural setting

Threats to Internal Validity • Time – History, Maturation • Measurement – Testing, Instrumentation • Selection – Randomization, Statistical regression, Mortality • Contamination – Diffusion, Compensation • Interaction among threats

Internal Validity • Establish that cause led to the effect – Identify non-experimental (uncontrolled) factors that affect the dependent variable or the relation b/w the independent and dependent variables

10 Threats to Internal Validity Time • History – Factors occurring along with the independent variable • An event occurring at the time of the intervention that overshadows (or reduces the effect of iv) and causes changes in dv – E. g. , change in manufacturing material along with training influences performance quality

Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4 E FIGURE 7. 1

10 Threats to Internal Validity Time • Maturation – Biological or psychological processes or factors that occur w/the passage of time & affect the dependent variable • E. g. , job experience, socialization etc. affect performance

Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4 E FIGURE 7. 2

10 Threats to Internal Validity Measurement • Testing – Studies w/pre-post measures or multiple measures • Increased or decreased sensitization to posttest measures • Changes in participants due to measurement itself • Increased fatigue or resentment • Self-consistency effects

10 Threats to Internal Validity • Instrumentation – Change in the measures from pre to post test • Change in observers’ frames of reference • Changes in participants’ frames of reference (e. g. , beta & gamma changes p. 43 Saks) • Change in instrumentation quality (e. g. , performance quality measurement)

10 Threats to Internal Validity • Instrumentation – Use of different measurement techniques • Establish expected convergent & discriminant validity • Importance of using different measures to overcome percept-percept problem

Threats to Internal Validity Selection • Randomization – Randomly assign to experimental conditions • Only in experimental designs • Matching on key variable if random assignment not possible

Threats to Internal Validity Selection • Randomization – Randomly select from population • Calculate response rate • Establish comparability of respondents to population on key features • Stratified (v. cluster? ) sampling

10 Threats to Internal Validity Selection • Statistical Regression – Upon repeat measurement, scores usually “regress” or “move toward” the mean • Usually when extreme scorers are selected into the study – Can be classified as ‘error’ variation

10 Threats to Internal Validity Selection • Mortality – Features of participants who do not complete the study affect the results – Reduced power in testing effects

10 Threats to Internal Validity Contamination • Diffusion/imitation of treatments – When people in control groups inadvertently know about or are affected by intervention and that changes their behavior

10 Threats to Internal Validity Contamination • Compensatory Effects – Knowledge about intervention and changes in control group members’ behavior is more purposeful

10 Threats to Internal Validity • Interaction among threats – When multiple threats operate at the same time obscuring the effect of the IV on the DV

Threats to Internal Validity • Time-related – History, maturation • Measurement – Testing, Instrumentation • Selection – Randomization, Statistical regression, mortality • Contamination – Diffusion, Compensation • Interaction among threats