Chapter 8 Construct and External Validity in Experimental

Chapter 8 Construct and External Validity in Experimental Research Back to Brief Contents n Construct Validity n External Validity n Cautions in Evaluating the External Validity of Experiments n Relationship between Internal and External Validity ♣

8. 1 Construct Validity -1 Back to Chapter Contents n Definition—extent to which we can infer higher-order constructs for our operations Fig 8. 1 n Constructs are used for ¡ ¡ Research participants Independent variable Dependent variable Experimental setting

8. 1 Construct Validity -2 Back to Chapter Contents n Assessing Construct Validity n Threats to Construct Validity ¡ Reactivity to the Experimental Situation ¡ Experimenter Effect

Assessing Construct Validity -1 Back to Chapter Contents n Have a clear definition of the construct of interest ¡ Problem is identifying prototypical features of the constructs ¡ Affects ability to identify the concrete operations used to represent the construct

Assessing Construct Validity -2(end) Back to Chapter Contents n Assess the match between the constructs and the operations used to represent them ¡ Ways to assess n Content validity n Criterion-related validity n — Predictive — concurrent Any other source of evidence

Threats to Construct Validity Back to Chapter Contents n n n n Inadequate explanation of the construct Tab 8. 1 Construct confounding Mono-operation bias Mono-method bias Confounding constructs with level of constructs Treatment-sensitive factorial structure Reactive self-report changes Reactivity to the experimental situation Experimenter effects Novelty and disruption effects Compensatory equalization Compensatory rivalry Treatment diffusion

Threats to Construct Validity Back to Chapter Contents n Inadequate explanation of the construct If a construct is not adequately explained analyzed, it can lead to a set of operations that do not represent the construct adequately n Construct confounding the operations used in a study represent more than one construct n Mono-operation bias a study uses only one operationalization of a construct This typically results in an underrepresentation of the construct and lowers construct validity

Threats to Construct Validity Back to Chapter Contents n Mono-method bias a study uses only one method (e. g. , physiological recording) to operationalize a construct The method used may influence the results n Confounding constructs with level of constructs a study investigates only a few levels of a construct (e. g. , three doses of a drug), but makes inferences about the overall construct (e. g. , the overall effect of the dose) n Treatment-sensitive factorial structure an instrumentation change that occurs because of the experimental treatment

Threats to Construct Validity Back to Chapter Contents n Reactive self-report changes that a research participant may make on self-report measures as a result of a motivational shift after being included in the experimental study n Reactivity to the experimental situation research participants’ perceptions and motives can affect the responses they make to the dependent variable and these responses can be interpreted as part of the treatment construct being tested

Threats to Construct Validity Back to Chapter Contents n Experimenter effects the experimenter’s attributes and expectancies can influence the responses made by the research participants and these responses can be interpreted as part of the treatment construct being tested n Novelty and disruption effects Research participants usually respond better to a new and novel situation and poorly to one that disrupts their routine These effects are part of the overall treatment effect

Threats to Construct Validity Back to Chapter Contents n Compensatory equalization individuals try to provide the same benefits or services to the control group that are received by the experimental group n Compensatory rivalry individuals resent being assigned to the control group and respond more negatively than would be expected, because of the resentment they feel n Treatment diffusion individuals in one treatment group receive some or all of another group’s treatment

Reactivity to the Experimental Situation 1/3 Back to Chapter Contents n Refers to research participants’ motives and perceptions influencing the response to the DV n Participant Effect ¡ Demand characteristics Any of the cues available in an exp, such as instructions, experimenter, rumors, experimental settings ¡ Primary motive — positive self-presentation

Reactivity to the Experimental Situation 2/3 Back to Chapter Contents n Conditions producing a positive self-presentation motive ¡ Tedeschi, Schlenker, & Bonoma (1971) P believe that others view their behavior as indicative of their true intentions, beliefs, or feelings Exp. constructed P believe that others think their behavior is externally determined

Reactivity to the Experimental Situation 3/3 Back to Chapter Contents n Implication for research ¡ Intertreatment interaction Perception by P in different treatment groups that they can fulfill the positive self-presentation motive by responding in different ways ¡ Intratreatment interaction Perception by P in the same treatment group that they can fulfill the positive self-presentation motive by responding in different ways

Experimenter Effect 1/3 Back to Chapter Contents n Experimenter Effect Fig 8. 2 ¡ Experimenter has motive of supporting the study hypothesis ¡ Can unintentionally lead to recording errors

Experimenter Effect 2/3 Back to Chapter Contents n Ways experimenter may bias the study ¡ Experimenter attributes n Biosocial attributes (e. g. ) age, sex, race n Psychosocial attributes psychometrically determined characteristics (e. g. ) anxiety level, hostility, authoritarianism, intelligence, dominance, warmth n Situational factors (e. g. ) prior contact of P, naive or experienced ¡ Experimenter expectancies (cont. )

Experimenter Effect 3/3 Back to Chapter Contents n Ways experimenter may bias the study ¡ Experimenter attributes ¡ Experimenter expectancies n Effect on experimenter (e. g. ) recording errors (biased, but low error rate) n Effect on research participant n Mediation of expectancies n Tab 8. 2 — Handling in animal research — Nonverbal communication in human studies Magnitude of expectancies—can exist in animal and human research and can be greater than the IV Fig 8. 3

8. 2 External Validity 1/4 Back to Chapter Contents n Definition—generalizing across people, settings, treatment variations, outcomes and times n Represents a test of interactions n Threats to external validity ¡ Population validity ¡ Ecological validity ¡ Temporal validity ¡ Treatment variation validity ¡ Outcome validity

8. 2 External Validity 2/4 Back to Chapter Contents n Population Validity The extent to which the results of a study can be generalized to the larger population ¡ Two-step inferential process Fig 8. 4 Tab 8. 3 Sample Experimentally accessible population Target population ¡ College students as P less emotionally and impulsively in laboratory studies ¡ Selection by Treatment Interaction if interaction exists cannot generalize to the target population

8. 2 External Validity 3/4 Back to Chapter Contents n Ecological Validity The extent to which the results of a study can be generalized across settings or environmental conditions n Temporal Validity The extent to which the results of an experiment can be generalized across time ¡ Seasonal variation A variation that occurs at regular time intervals ¡ Cyclical variation A regular variation that occurs within people and other organisms (ex) circadian rhythm, MC

8. 2 External Validity 4/4 Back to Chapter Contents n Treatment Variation Validity (IV) The generalizability of results across variation of the treatment n Outcome Validity (DV) The generalizability of results across different but related dependent variables

8. 3 Cautions in Evaluating the External Validity of Experiments Back to Chapter Contents n Some studies conducted to increase knowledge and not to generalize to a real-life situation n Some studies assess a theoretical process n Moving out of the laboratory does not insure generalization

8. 4 Relationship between Internal and External Validity Back to Chapter Contents n Relationship between internal and external validity —often an inverse relationship