Validity of Quantitative Research Conclusions Validity of Quantitative

Validity of Quantitative Research Conclusions

Validity of Quantitative Research Conclusions Internal Validity Issues of Cause and Effect External Validity Issues of Generalizability

Validity of Quantitative Research Conclusions Issues of Cause and Effect Internal Validity Issues of Generalizability External Validity

Validity of Quantitative Research Conclusions Issues of Cause and Effect Statistical Conclusion Validity Internal Validity Issues of Generalizability Construct Validity External Validity

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions

Statistical Conclusion Validity Is there a relationship between A & B? Threat Low Statistical Power Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects Solutions

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions

Internal Validity Is there a cause and effect relationship between A & B? Threat History Maturation Testing Instrumentation Statistical Regression (to the Mean) Selection (and Interaction with Selection) Mortality Unknown Direction of Causation Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Solutions

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Poor Representation of Constructs Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables Solutions

External Validity Is the relationship between A and B generalizable? Threat Solutions

External Validity Is the relationship between A and B generalizable? Threat Sample does not represent the population of interest. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction) Solutions

Research Design Choices • • • • Use large sample size. Choose powerful statistics tests. Meet assumptions of statistics. Choose robust statistics tests. Use a conservative post-hoc comparison correction procedure. Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Precisely define & standardize treatment. Choose settings free of confounds. Use more than one dependent variable. Select subjects randomly from a welldefined population. Vary settings and analyze across settings. Replicate the same experiment at different times. Conduct a literature review to see if prior evidence is consistent with hypotheses. • • • • Measure potential confounds. Use a within-subjects (repeated Measures design. Randomly assign to treatment group. Share study rationale with subjects. Treat all subjects justly. Hide true hypotheses from subjects. Define constructs precisely. Use measures with high validity. Measure dependent variable with multiple measures. Manipulate independent variable in many ways. Use double-blind study design. Use continuous level of measurement. Give a single treatment to subjects. Measure effect of pretests on dependent variable.

Statistical Conclusion Validity Is there a relationship between A & B? Threat Low Statistical Power Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects Solutions

Statistical Conclusion Validity Is there a relationship between A & B? Threat Low Statistical Power Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects Solutions Use large sample size. Choose powerful statistics tests.

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Choose settings free of confounds. Measure potential confounds. Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? Threat Solutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Choose settings free of confounds. Measure potential confounds. Random Differences in Subjects Use a within-subjects (repeated measures) design.

Internal Validity Is there a cause and effect relationship between A & B? Threat History Maturation Testing Instrumentation Statistical Regression (to the Mean) Selection (and Interaction with Selection) Mortality Unknown Direction of Causation Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Solutions

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Hide True Hypotheses from Subjects

Internal Validity Is there a cause and effect relationship between A & B? Threat Solutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Hide True Hypotheses from Subjects Treat All Subjects Justly

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Poor Representation of Constructs Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables Solutions

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Poor Representation of Constructs Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables Solutions Define constructs precisely. Use measures with high validity.

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Use double-blind study design. Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Use double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Use double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment Confound Give a single treatment to subjects, not a mix. Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Use double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment Confound Give a single treatment to subjects, not a mix. Measurement and Treatment Interaction Measure effects of pretests on dependent variable. Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? Threat Solutions Poor Representation of Constructs Define constructs precisely. Use measures with high validity. Single Operation Bias Measure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Measure dependent variable with multiple methods. Hypothesis Guessing by Subjects Hide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Use measures with high validity. Experimenter Expectations Use double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment Confound Give a single treatment to subjects, not a mix. Measurement and Treatment Interaction Measure effects of pretests on dependent variable. Restricted Generalization Across Possible Dependent Variables Use more than one dependent variable.

External Validity Is the relationship between A and B generalizable? Threat Sample does not represent the population of interest. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction) Solutions

External Validity Is the relationship between A and B generalizable? Threat Sample does not represent the population of interest. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction) Solutions Select subjects randomly from a well-defined population.

External Validity Is the relationship between A and B generalizable? Threat Solutions Sample does not represent the population of interest. Select subjects randomly from a well-defined population. The setting of a study does not represent all settings of interest. Vary settings and analyze for relationships in each setting. The time during which a study takes place does not represent all future times. (History and Treatment Interaction)

External Validity Is the relationship between A and B generalizable? Threat Solutions Sample does not represent the population of interest. Select subjects randomly from a well-defined population. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction) Vary settings and analyze for relationships in each setting. Replicate the same experiment at different times. Conduct a literature review to see if prior evidence is consistent with hypotheses.