Hypothesis Testing Validity and Threats to Validity Graziano

  • Slides: 24
Download presentation
Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8

Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program. Copyright © Allyn & Bacon (2010)

Testing Hypotheses n Researchers tests specific hypotheses – Generated from the initial research idea

Testing Hypotheses n Researchers tests specific hypotheses – Generated from the initial research idea through a series of steps n A research idea can stimulate dozens of research hypotheses depending on how – it is translated into a statement of the problem – the variables are operationally defined Copyright © Allyn & Bacon (2010)

The Initial Idea n The initial idea is the starting point – Often vague

The Initial Idea n The initial idea is the starting point – Often vague or general – It requires refining before research hypotheses can be generated n Refinement of the initial idea is based on (1) a search of relevant research literature (2) initial observations of the phenomenon n Narrow and formalize the initial idea into a statement of the problem Copyright © Allyn & Bacon (2010)

Statement of the Problem n In the form of a question – Clearly indicates

Statement of the Problem n In the form of a question – Clearly indicates an expected relationship – Nature of the question dictates the required level of constraint of a study n n n Causal questions will require experimental research Questions about relationships can be answered with lower-constraint research Convert into research hypotheses by operationally defining the variables Copyright © Allyn & Bacon (2010)

Generating Research Hypotheses n Ideas lead to – observations – library research n n

Generating Research Hypotheses n Ideas lead to – observations – library research n n Statement of problem Problem statements become research hypotheses when constructs are operationalized Copyright © Allyn & Bacon (2010)

Operational Definitions Initially covered in Chapter 3 n The procedures used to measure and/or

Operational Definitions Initially covered in Chapter 3 n The procedures used to measure and/or manipulate variables n Most variables can be operationally defined in many different ways n Copyright © Allyn & Bacon (2010)

Research Hypotheses State clearly the expected relationship between the variables n The form is

Research Hypotheses State clearly the expected relationship between the variables n The form is a declarative statement, but it is a tentative statement to be tested in research n Variables in research hypotheses are stated in operational definition terms n Copyright © Allyn & Bacon (2010)

The Role of Theory n Theory guides all research planning – Often the primary

The Role of Theory n Theory guides all research planning – Often the primary source of the research hypothesis – Guides the selection of variables – Guides the operational definitions of variables n Most research is based on multiple, overlapping, and interacting theories Copyright © Allyn & Bacon (2010)

Testing Research Hypotheses n Actually testing three sets of hypotheses – – – n

Testing Research Hypotheses n Actually testing three sets of hypotheses – – – n The null hypothesis The confounding variable hypotheses The causal hypothesis Accept causal hypothesis only if you – reject null hypothesis (statistical analysis) – rule out each potential confounding variable hypothesis (based on appropriate controls) Copyright © Allyn & Bacon (2010)

Evaluating Hypotheses Copyright © Allyn & Bacon (2010)

Evaluating Hypotheses Copyright © Allyn & Bacon (2010)

Types of Validity n Statistical Validity n Construct Validity n External Validity n Internal

Types of Validity n Statistical Validity n Construct Validity n External Validity n Internal Validity Copyright © Allyn & Bacon (2010)

Statistical Validity Are the statistical tests accurate? n Threatened by n – Unreliable measures

Statistical Validity Are the statistical tests accurate? n Threatened by n – Unreliable measures – Violations of statistical assumptions n Strengthened by – Using well validated measures – Having approximately equal sample sizes in each group (covered in Appendix D) Copyright © Allyn & Bacon (2010)

Construct Validity n n Is our theory the best explanation for the results? Threatened

Construct Validity n n Is our theory the best explanation for the results? Threatened by – Any alternative explanation for the results n Strengthened by – Using well-validated constructs to build theoretical predictions for the study Copyright © Allyn & Bacon (2010)

External Validity n n Do the results apply to the broader population? Threatened by

External Validity n n Do the results apply to the broader population? Threatened by – Unrepresentative samples – Generalizing beyond the limits of the sample n Strengthened by – Gathering a representative sample (if possible) – Clearly describing sample, so that other researchers will know the limits of generalization Copyright © Allyn & Bacon (2010)

Internal Validity Is the independent variable responsible for the observed changes in the dependent

Internal Validity Is the independent variable responsible for the observed changes in the dependent variable? n Threatened by n – Confounding variables n Strengthened by – Adding adequate controls to reduce or eliminate confounding Copyright © Allyn & Bacon (2010)

Avoiding Confounding n Confounding and internal validity – Many sources for confounding (covered next)

Avoiding Confounding n Confounding and internal validity – Many sources for confounding (covered next) – With proper controls, confounding can be virtually eliminated (see Chapter 9) n Confounding and construct validity – Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon Copyright © Allyn & Bacon (2010)

Confounding Variables 1 n Maturation – Changes due to growth or predictable changes n

Confounding Variables 1 n Maturation – Changes due to growth or predictable changes n History – Changes due to an event that occurs during the study n Testing – Changes due to the effects of previous testing Copyright © Allyn & Bacon (2010)

Confounding Variables 2 n Instrumentation – Any change in the calibration of the measuring

Confounding Variables 2 n Instrumentation – Any change in the calibration of the measuring instrument over the course of the study n Regression to the Mean – Tendency for participants selected because of extreme scores to be less extreme on a retest n Selection – Any factor that creates groups that are not equal at the start of the study Copyright © Allyn & Bacon (2010)

Confounding Variables 3 n Attrition – Loss of participants during a study; are the

Confounding Variables 3 n Attrition – Loss of participants during a study; are the participants who drop out different from those who continue? n Diffusion of treatment – Changes in participants’ behavior due to information they obtained about other conditions n Sequence Effects – Effects on performance in one condition due to experience with previous conditions Copyright © Allyn & Bacon (2010)

Subject Effects n Participants are not passive – They try to understand the study

Subject Effects n Participants are not passive – They try to understand the study to help them to know what they “should do” (termed subject effects) – Respond to subtle cues about what is expected (termed demand characteristics) n Placebo effect: treatment effect due to expectations that the treatment will work Copyright © Allyn & Bacon (2010)

Experimenter Effect n n Based on the expectations of the researcher Can affect the

Experimenter Effect n n Based on the expectations of the researcher Can affect the outcome of studies if not controlled May be due to the experimenter providing demand characteristics to the participant Not the same as scientific fraud (which is deliberate) Copyright © Allyn & Bacon (2010)

Validity, Control, and Constraint Three closely-tied concepts n Validity n – The accuracy of

Validity, Control, and Constraint Three closely-tied concepts n Validity n – The accuracy of the study or procedure – Increased by using appropriate control procedures n The more controls we employ, the higher the level of constraint Copyright © Allyn & Bacon (2010)

Ethical Principles n Risk is balanced by reward – A poorly designed study will

Ethical Principles n Risk is balanced by reward – A poorly designed study will provide no useful information; therefore, any risk would be unacceptable n Informed Consent – Virtually guarantees that you will have confounding due to selection because some people will refuse to participate – A small price to pay to maintain ethical standards Copyright © Allyn & Bacon (2010)

Summary n n Start by building a research hypothesis Testing the research hypothesis is

Summary n n Start by building a research hypothesis Testing the research hypothesis is actually testing three hypotheses – (1) null; (2) confounding-variable; (3) causal n n n Several types of validity Many potential confounding variables Subject and experimenter effects can also affect the outcome of the study Copyright © Allyn & Bacon (2010)