EXPERIMENTS THREATS TO VALIDITY Overview Experimental Research Definition

EXPERIMENTS & THREATS TO VALIDITY

Overview Experimental Research � Definition and terms � Characteristics � Types Definition of internal & external validity � Threats to internal validity � Threats to external validity � Threats to both

Validity of a Study

Validity (of a study) Internal validity � Is there only one unambiguous explanation? Have you eliminated confounds? Can the changes in the DV be attributed to the manipulation of IV? External validity � Do the results generalize? Can the results apply to populations, situations, measurement procedures which were not part of original study?

Threats to Internal Validity Confound variables 1. Threats for all studies: Environmental variables � E. g. , Time of day, room size, room tempt, noise 2. Threats when comparing groups � Assignment bias (e. g. , non-equivalent control group) Groups are not equal (e. g. , personality characteristics or experiences)

Threats to Internal Validity 3. Threats over time � History Outside events occur � Maturation Systematic changes in people (e. g. , age) � Instrumentation Changes � Diffusion in measurement, scales, or criteria of treatment Communication across groups can interfere with manipulation of IV

Threats to Internal Validity 3. Threats over time (continued) � Testing effects (can happen outside withinsubjects designs too) Changes due to testing Carry-over effects: fatigue, practice, and context effects � Statistical regression Regression towards the mean extreme scores become less extreme � Experimental Differential Mortality (i. e. , attrition) dropout rates between groups

Threats to External Validity 1. Generalizing across participants � Selection Bias bias in sampling procedure � College students � Volunteer bias � Participant characteristics � Cross-species generalization

Threats to External Validity 2. Generalizing across features of study � Novelty Do participants respond differently in the lab? � Multiple Did effects treatment effects participation in earlier treatment affect outcomes? � Experimenter Would characteristics we get the same results if we used a different experimenter?

Threats to External Validity 3. Generalization across features of measures � Sensitization Assessment influences participants Makes them more aware of thoughts, feelings, or behavior � Time of measurement Would we get the same result if we measured 1 week after intervention vs. 5 weeks?

Balancing Internal & External Validity We want both, but usually a trade-off � Higher internal validity associated w/ greater control � Greater control associated w/ lower external validity We have to decide what’s most important � Depends on purpose of study Another reason why it is the accumulation of information that is important! (i. e. , multiple

Threats to internal & external validity Experimental bias (i. e. , expectancy effect) � Experimenter expectations influence results � Not necessarily intentional

Threats to internal & external validity Reactivity � Participants change behavior because they know they are participating in a study Demand characteristics � Cues make hypothesis obvious � Influence participants’ responses/behavior (e. g. , subject effect) � The ‘good’ participant � Apprehensive attitude � The ‘negative’ participant

Reducing bias, reactivity & other issues Standardize or automate experiment Blind � Single- or double-blind studies Conducting a preliminary study

Reducing bias, reactivity, & other issues Cover stories Add other unrelated DVs Use involving activities Habituation Indirect measures Using placebos