Threats to Validity Threats to validity definition Theory

Threats to Validity • Threats to validity – definition • Theory vs. measurement • Types of validity – – Conclusion validity Internal validity Construct validity External validity • Particular threats to validity Validity

Threats to validity - definition • Researchers do not simply • The description is not like report facts. a photograph – it is not “veridical. ” It can be wrong in a variety of ways. • The researcher’s task is • Problems that cause our like describing an object descriptions to be wrong covered by a thick blanket. are problems of validity. Validity

Theory vs. measurement • Our interest is always in the validity of a conclusion or inference Theory • That validity can be threatened either by poor measurement or by poor theory Validity Measurement

Theory vs. measurement • Our theory predicts that we will observe a relationship between measurable variables (that is, variables we have operationalized). • We won’t see the expected relationship if Validity – theory is incorrect, or – the operationalizations are flawed.

Cause Effect Independent Variable Validity Dependent Variable

Theory vs. measurement • Suppose you believe that absence makes the heart grow fonder. • Your theory has two constructs: – Absence – Fondness. • Both must be ‘operationalized. ’ Validity

Absence Fondness # days apart # phone calls We might not see the expected relationship because our theory is wrong operationalizations are poor. Validity

Types of Validity • Does our study have: – Is the relationship between variables seen in the data the one that our theory claims should be there? – conclusion validity? Validity

Types of Validity • Does our study have: – Is the relationship between variables causal? (A B) – conclusion validity? – internal validity? Validity

Types of Validity • Does our study have: • – conclusion validity? – internal validity? – construct validity? Does the I. V. capture our theoretical cause? • Does the D. V. capture our theoretical effect? Validity

Types of Validity • Does our study have: – Will the findings in our study be replicated in other settings? – conclusion validity? – internal validity? – construct validity? – internal validity? Validity

Conclusion Validity I. V. D. V. Validity

Internal Validity • If there is some confounding variable – one that varies # days apart with our I. V. – then our study lacks internal validity # phone calls money left Validity

Construct Validity Cause Effect I. V. D. V. Validity

Absence Fondness Distance A different operationalization might produce different conclusions – e. g, if distance varies with the purpose of the trip, which in turn determines how busy the traveler is through the day. Validity # thoughts about loved one

External Validity Cause Effect Validity

Particular threats to validity • On previous slides, we’ve discussed several types of validity. • The next slides list some threats to each type. • You should be able to discuss and give examples of each of these threats. Validity

Threats to Internal Validity • History • changes happening to only one of two or more groups that are compared Validity

Threats to Internal Validity • • History Maturation • changes in performance associated with development Validity

Threats to Internal Validity • • • History Maturation Test practice • changes in performance due to experience with the test format Validity

Threats to Internal Validity • • History Maturation Test practice Regression to the mean • extreme values of measurements are unlikely to be repeated Validity

Threats to Internal Validity • • • History Maturation Test practice Regression to the mean Selection • between-group differences that are present at the start Validity

Threats to Internal Validity • • • History Maturation Test practice Regression to the mean Selection Mortality • differences produced by different rates of dropout between groups Validity

Threats to Conclusion validity • Conclusion validity may be low because of: – poor reliability • increase number of items used in the task • or use more precise measuring instruments Validity

Threats to Conclusion validity • Conclusion validity may be low because of: – poor reliability – weak relationship • increase strength by increasing “dose” • decrease “noise” by increasing reliability or decreasing distractions in the environment Validity

Threats to Conclusion validity • Conclusion validity may be low because of: • Increase number of observations, by – increasing number of subjects – increasing number of trials per subject – poor reliability – weak relationship – lack of statistical power Validity

Threats to Construct Validity • Loose connection between theory and method • Ambiguous effect of independent variables – Hawthorne effect – Good subject tendency – Evaluation apprehension (social desirability, white coat syndrome) Validity

Threats to Construct Validity • Interaction of different treatments • Interaction of testing and treatment. Validity

Threats to External Validity • Any dimension on which studies can differ might produce a failure to replicate. • Our subjects might be special in some way – Other people Validity

Threats to External Validity • Any dimension on which people can differ might produce a failure to replicate. – Other people – Other places • our environment might be special in some way • especially if we work in a lab. Validity

Threats to External Validity • Any dimension on which people can differ might produce a failure to replicate. – Other people – Other places – Other times • our time might be special in some way • e. g. , right after a government report on smoking is released, you survey smokers Validity

Challenge • For each of the following maxims, suggest some operationalization of the relevant constructs: • Too many cooks spoil the broth • Many hands make light work • You snooze, you lose • A rolling stone gathers no moss Validity