Hypothesis Testing Validity and Threats to Validity Graziano

Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program. Copyright © Allyn & Bacon (2007)

Testing Hypotheses �Research tests specific hypotheses ◦ Generated from the initial research idea through a series of steps �A research idea can generate dozens of research hypotheses depending on how ◦ it is translated into a statement of the problem ◦ the variables are operationally defined Copyright © Allyn & Bacon (2007)

The Initial Idea �The initial idea is the starting point ◦ Often vague or general ◦ It requires refining before research hypotheses can be generated �Refinement of the initial idea is based on (1) a search of relevant research literature (2) initial observations of the phenomenon �Narrow and formalize the initial idea into a statement of the problem Copyright © Allyn & Bacon (2007)

Statement of the Problem �In the form of a question ◦ Clearly indicates an expected relationship ◦ Nature of the question dictates the required level of constraint of a study �Causal questions will require experimental research �Questions about relationships can be answered with lower -constraint research �Convert into research hypotheses by operationally defining the variables Copyright © Allyn & Bacon (2007)

Generating Research Hypotheses Ideas lead to ◦ observations ◦ library research Statement of problem Problem statements become research hypotheses when constructs are operationalized Copyright © Allyn & Bacon (2007)

Operational Definitions � The procedures used to measure and/or manipulate variables � Most variables can be operationally defined in many different ways Copyright © Allyn & Bacon (2007)

Research Hypotheses � State clearly the expected relationship between the variables � The form is a declarative statement, but it is a tentative statement to be tested in research � Variables in research hypotheses are stated in operational definition terms Copyright © Allyn & Bacon (2007)

The Role of Theory to the research Hypothesis � Theory guides all research planning ◦ Often the primary source of the research hypothesis ◦ Guides the selection of variables ◦ Guides the operational definitions of variables � Most research is based on multiple, overlapping, and interacting theories Copyright © Allyn & Bacon (2007)

Testing Research Hypotheses �Actually testing three hypotheses ◦ The null or statistical hypothesis ◦ The confounding variable hypothesis ◦ The causal hypothesis �Accept causal hypothesis only if you ◦ reject null hypothesis (statistical analysis) ◦ rule out each potential confounding variable hypothesis (based on appropriate controls) Copyright © Allyn & Bacon (2007)

Evaluating Hypotheses Copyright © Allyn & Bacon (2007)

Types of Validity �Statistical Validity �Construct Validity �External Validity �Internal Validity Copyright © Allyn & Bacon (2007)

Statistical Validity � Are the statistical tests accurate? � Threatened by ◦ Unreliable measures ◦ Violations of statistical assumptions � Strengthened by ◦ Using well validated measures ◦ Having approximately equal sample sizes in each group (More details about choosing the right sample will be discussed later). Copyright © Allyn & Bacon (2007)

Construct Validity �Is our theory the best explanation for the results? �Threatened by ◦ Any alternative explanation for the results �Strengthened by ◦ Using well-validated constructs to build theoretical predictions for the study Copyright © Allyn & Bacon (2007)

External Validity �Do the results apply to the broader population? �Threatened by ◦ Unrepresentative samples ◦ Generalizing beyond the limits of the sample �Strengthened by ◦ Gathering a representative sample (if possible) ◦ Clearly describing sample, so that other researchers will know the limits of generalization Copyright © Allyn & Bacon (2007)

Internal Validity � Is the independent variable responsible for the observed changes in the dependent variable? � Threatened by ◦ Confounding variables � Strengthened by ◦ Adding adequate controls to reduce or eliminate confounding Copyright © Allyn & Bacon (2007)

Avoiding Confounding �Confounding and internal validity ◦ Many sources for confounding (covered next) ◦ With proper controls, confounding can be virtually eliminated ◦ Confounding and construct validity ◦ Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon Copyright © Allyn & Bacon (2007)

Confounding Variables 1 � Maturation ◦ Changes due to growth or predictable changes � History ◦ Changes due to an event that occurs during the study � Testing ◦ Changes due to the effects of previous testing Copyright © Allyn & Bacon (2007)

Confounding Variables 2 � Instrumentation ◦ Any change in the calibration of the measuring instrument over the course of the study � Regression to the Mean ◦ Tendency for participants selected because of extreme scores to be less extreme on a retest � Selection ◦ Any factor that creates groups that are not equal at the start of the study Copyright © Allyn & Bacon (2007)

Confounding Variables 3 � Attrition ◦ Loss of participants during a study; are the participants who drop out different from those who continue? � Diffusion of treatment � Sequence Effects ◦ Changes in participants’ behavior due to information they obtained about other conditions ◦ Effects on performance in one condition due to experience with previous conditions Copyright © Allyn & Bacon (2007)

Subject Effects �Participants are not passive ◦ They try to understand the study to help them to know what they “should do” (termed subject effects) ◦ Respond to subtle cues about what is expected (termed demand characteristics) �Placebo effect: treatment effect due to expectations that the treatment will work Copyright © Allyn & Bacon (2007)

Experimenter Effect �Based on the expectations of the researcher �Affects the outcome of studies if not controlled �May be due to the experimenter providing demand characteristics to the participant �Not the same as scientific fraud (which is deliberate) Copyright © Allyn & Bacon (2007)

Validity, Control, and Constraint � Three closely-tied concepts � Validity ◦ The accuracy of the study or procedure ◦ Increased by using appropriate control procedures � The more controls we employ, the higher the level of constraint Copyright © Allyn & Bacon (2007)

Summary �Start by building a research hypothesis �Testing the research hypothesis is actually testing three hypotheses ◦ (1) null; (2) confounding-variable; (3) causal �Several types of validity �Many potential confounding variables �Subject and experimenter effects can also affect the outcome of the study Copyright © Allyn & Bacon (2007)