Chapter 11 Enhancing Rigor in Quantitative Research Copyright

Types of Validity • Statistical conclusion validity • Internal validity • Construct validity •

Controlling Intrinsic Source of Extraneous Variability • Randomization • Crossover • Homogeneity • Blocking/Stratification

Benefits and Limitations of Control Methods: Randomization Benefits: • Controls all extraneous variables •

Benefits and Limitations of Control Methods: Crossover (Repeated Measures) Benefits: • If done with

Benefits and Limitations of Control Methods: Homogeneity Benefits: • Easy to achieve • Enhances

Benefits and Limitations of Control Methods: Blocking Benefits: • Enhances interpretability of relationships •

Benefits and Limitations of Control Methods: Matching Benefits: • Enhances interpretability of relationships •

Benefits and Limitations of Control Methods: Statistical Studies Benefits: • Enhances interpretability of relationships

Threats to Statistical Conclusion Validity • Low statistical power • Low precision • Factors

Threats to Internal Validity • Temporal ambiguity • Selection • History • Maturation •

Research Designs and Threats to Internal Validity Copyright © 2008 Wolters Kluwer Health |

Construct Validity • Concerns inferences from the particular exemplars of a study to the

Threats to Construct Validity • Reactivity to the study situations • Researcher expectancies •

External Validity • Concerns inferences about the extent to which relationships observed in a

Threats to External Validity • Interaction between relationship and people • Interaction between causal

Accessible vs. Target Population • Accessible population: The population available for a particular study

Threats to External Population • Inadequate sampling • Expectancy effects • Novelty effects •

Slides: 18

Download presentation

Controlling Intrinsic Source of Extraneous Variability • Randomization • Crossover • Homogeneity • Blocking/Stratification • Matching (pair matching) • Statistical control Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Randomization Benefits: • Controls all extraneous variables • Does not require advance knowledge of which variables to control Limitations: • Ethical and practical constraints on manipulation • Possible artificiality of conditions Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Crossover (Repeated Measures) Benefits: • If done with randomization, strongest possible approach • Reduces sample size requirements Limitations: • Cannot be used if there are possible carryover effects from one condition to another Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Homogeneity Benefits: • Easy to achieve • Enhances interpretability of relationships Limitations: • Limits generalizability • Requires knowledge of which variables to control Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Blocking Benefits: • Enhances interpretability of relationships • Offers possibility of examining blocking variable as an independent variable Limitations: • Manageable only with a few blocking variables • Requires knowledge of which variables to control Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Matching Benefits: • Enhances interpretability of relationships • Easy to do if there is a large pool of available comparison group subjects Limitations: • Manageable only with a few matching variables • Requires knowledge of which variables to control • May be hard to find comparison group matches Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Benefits and Limitations of Control Methods: Statistical Studies Benefits: • Enhances interpretability of relationships • Easy and economical • Can be used with a large number of extraneous variables Limitations: • Requires knowledge of which variables to control • Requires statistical sophistication Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Threats to Statistical Conclusion Validity • Low statistical power • Low precision • Factors that undermine a strong operationalization of the independent variables • Inadequate participation in treatment conditions Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Threats to Internal Validity • Temporal ambiguity • Selection • History • Maturation • Mortality/Attrition • Testing and Instrumentation Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Construct Validity • Concerns inferences from the particular exemplars of a study to the higher order constructs that they are intended to represent Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Threats to Construct Validity • Reactivity to the study situations • Researcher expectancies • Novelty effects • Compensatory effects • Treatment diffusion or contamination Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

External Validity • Concerns inferences about the extent to which relationships observed in a study hold true over variations in people, conditions, and setting, as well as over variations in treatments and outcomes Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Threats to External Validity • Interaction between relationship and people • Interaction between causal effects and treatment variation Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Accessible vs. Target Population • Accessible population: The population available for a particular study • Target population: The total group of people in whom a researcher is interested and to whom results could be generalized Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins

Threats to External Population • Inadequate sampling • Expectancy effects • Novelty effects • Interaction of history and treatment effects • Experimenter effects • Measurement effects Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins