Chapter 6 Research Validity Research Validity Truthfulness of

  • Slides: 45
Download presentation
Chapter 6 Research Validity

Chapter 6 Research Validity

Research Validity: Truthfulness of inferences made from a research study.

Research Validity: Truthfulness of inferences made from a research study.

Statistical Conclusion Validity: Valid statements of the covariation (relationship) between the Independent and Dependant

Statistical Conclusion Validity: Valid statements of the covariation (relationship) between the Independent and Dependant Variable. http: //www. nutrisystem. c om/jsps_hmr/landingpage s/select-new. jsp e. g. , Heterogeneous results – present atypical cases.

Statistical Conclusion Validity: Using Statistics as evidence for the null hypothesis. Failing to reject

Statistical Conclusion Validity: Using Statistics as evidence for the null hypothesis. Failing to reject the null does not mean there is no difference between conditions anymore than failing to convict a defendant means that they are innocent. The study may not have had enough POWER to find differences.

Construct Validity Is the construct adequately represented by the measures used in the research

Construct Validity Is the construct adequately represented by the measures used in the research study?

Participant Reactivity Participant Motivations and tendencies.

Participant Reactivity Participant Motivations and tendencies.

Demand Characteristics Any cues in the experiment (e. g. , instructions, rumors, others behaviors

Demand Characteristics Any cues in the experiment (e. g. , instructions, rumors, others behaviors that influence the responses of the participants. e. g. , Yawn Study Subliminal Perception study - Positive self presentation - “screw you” attitude!

A. Experimenter Effects Actions and characteristics of researcher that influence the responses of participants.

A. Experimenter Effects Actions and characteristics of researcher that influence the responses of participants.

B. Experimenter Expectancies • Recording Errors • Interpretation Biases • Unintentionally treating one group

B. Experimenter Expectancies • Recording Errors • Interpretation Biases • Unintentionally treating one group differently than another.

C. Experimenter Attributes • Biosocial (age, sex, race etc. )

C. Experimenter Attributes • Biosocial (age, sex, race etc. )

Controlling for Demand Characteristics Can’t get rid of demand characteristics. • Withhold the hypotheses

Controlling for Demand Characteristics Can’t get rid of demand characteristics. • Withhold the hypotheses until after the study. • Distract participants from real variables. e. g. , asked participants to rate the commercials for emotions etc. • Do not tell participant which condition they are in. BE SURE DEMAND CHARACTERISTS DO NOT VARY ACROSS CONDITIONS!!

Internal Validity - The extent to which all explanations for changes in the DV

Internal Validity - The extent to which all explanations for changes in the DV between conditions have been eliminated -- other than the IV. ie(7 a) 12

Extraneous variable - any variable other than IV that effects DV. Control Extraneous Variable

Extraneous variable - any variable other than IV that effects DV. Control Extraneous Variable Experimental ie(7 a) 13

Control Confound Experimental Confounding Extraneous Variable - any extraneous variable that affects one condition

Control Confound Experimental Confounding Extraneous Variable - any extraneous variable that affects one condition differently than it affects other conditions. ie(7 a) 14

Between Subjects Designs -Two (or more) Groups of Participants compared to each other. Each

Between Subjects Designs -Two (or more) Groups of Participants compared to each other. Each group has a different level of the IV. Experimental (Treatment) Group Control Group ie(7 a) 15

Major Potential Confound: Individual Differences (I. e. , perhaps the participants in one group

Major Potential Confound: Individual Differences (I. e. , perhaps the participants in one group are not comparable in many ways to participants in the other group). ie(7 a) 16

Within Subjects Design One Group of participants measured under more than one condition of

Within Subjects Design One Group of participants measured under more than one condition of the IV. Experimental Condition Control Condition Potential Confounds Since subjects cannot be in more than one condition at one time, anything that is not the same at each of the times of measurement (other than the IV) is a potential confound. ie(7 a) 17

Pre-Post Test Designs (type of WS design) Pre-test - DV measured before Treatment -

Pre-Post Test Designs (type of WS design) Pre-test - DV measured before Treatment - serves as baseline control condition Post-test - DV measured after Treatment ie(7 a) 18

Other than the treatment (IV) what could cause difference in DV between conditions (I.

Other than the treatment (IV) what could cause difference in DV between conditions (I. e. , what possible confounds could there be? Confounds with Pre-Post Designs History - any changes that occurred between Pre and Post Test other than IV. ie(7 a) 19

Maturation - biological/psychological changes between pre-post test. Instrumentation - changes in measurement device or

Maturation - biological/psychological changes between pre-post test. Instrumentation - changes in measurement device or operational definition between pre and post test. ie(7 a) 20

Are rates of Diabetes really increasing, or are we simply diagnosing more cases? ?

Are rates of Diabetes really increasing, or are we simply diagnosing more cases? ? ? Attention Deficit Disorder? ie(7 a) 21

Testing : Changes is a person’s score for the post conditions that results from

Testing : Changes is a person’s score for the post conditions that results from having been tested in the pre-test. - Sensitization - Boredom - Practice ie(7 a) 22

Attrition - subjects dropping out of the study. ANYTHING that is a difference between

Attrition - subjects dropping out of the study. ANYTHING that is a difference between the before and after condition other than the IV is a confound. ie(7 a) 23

Hokey Pokey ie(7 a) 24

Hokey Pokey ie(7 a) 24

Artifact - Effect caused by the procedure rather than by the IV. Statistical Regression

Artifact - Effect caused by the procedure rather than by the IV. Statistical Regression Artifact. - problem when subjects are assigned to groups based on the pre-test scores. - group scores will be pulled towards the mean of the DV. - Pre-test high scorers will appear to do poorer. Pre-test low scorers will appear to do better. - problem with unreliable measures ie(7 a) 25

Not a problem if the DV is very reliable. ie(7 a) 26

Not a problem if the DV is very reliable. ie(7 a) 26

Internal Validity? Confounds (in pre and post designs) • History • Maturation • Instrumentation

Internal Validity? Confounds (in pre and post designs) • History • Maturation • Instrumentation • Testing • Attrition (Mortality) • Statistical Regression Artifact* ie(7 a) 27

Controlling for these confounds. Use a pre-post control group. Treated the same as treatment

Controlling for these confounds. Use a pre-post control group. Treated the same as treatment Group except IV is not manipulated between tests. Control group affected by same, history, Maturation and Regression effects. Any difference between the treatment and control groups are not due to confounds. ie(7 a) 28

Experimental Group Pre-test Treatment plus confounds Control Group Pre-test confounds ie(7 a) Post-test 29

Experimental Group Pre-test Treatment plus confounds Control Group Pre-test confounds ie(7 a) Post-test 29

Selection – confound due to assignment of subjects to the Control and Treatment Groups.

Selection – confound due to assignment of subjects to the Control and Treatment Groups. Compare Pre-test scores. Are they the same to begin with? But could be a difference that interacts with the treatment. - use random assignment!!!! ie(7 a) 30

Selection Confound -subjects assigned to treatment and control group biased on a criteria (bias).

Selection Confound -subjects assigned to treatment and control group biased on a criteria (bias). Selection X (interactions) Any of the five confounds effect the control group differently than the treatment group. i. e. , Alcohol treatment study - compare volunteers to non-volunteers e. g. , Differential History ie(7 a) 31

Selection X (interactions) Any of the five confounds effect the control group differently than

Selection X (interactions) Any of the five confounds effect the control group differently than the treatment group. Differential History Differential Attrition i. e. , Alcohol treatment study - compare volunteers to non-volunteers ie(7 a) 32

Differential Attrition Weight loss Study (Diet and Exercise condition) No-Treatment Control Group Type of

Differential Attrition Weight loss Study (Diet and Exercise condition) No-Treatment Control Group Type of people that drop put of the study might depend on which study they are in. ie(7 a) 33

How can we ensure the subjects in each condition are comparable? Random Assignment Extraneous

How can we ensure the subjects in each condition are comparable? Random Assignment Extraneous variables still effect DV, but it should not be a confound. ie(7 a) 34

Pre-test Post test Treatment Control Treatment group = size of treatment effect plus confounds

Pre-test Post test Treatment Control Treatment group = size of treatment effect plus confounds Control group = estimate of the size of prepost confounds. ie(7 a) 35

Pre-test Post test Treatment Control Is the pre-post test change significantly greater in the

Pre-test Post test Treatment Control Is the pre-post test change significantly greater in the Treatment Condition than in the control condition? ie(7 a) 36

External Validity Goal of Psychology: to determine the Underlying “laws” of behavior. External Validity

External Validity Goal of Psychology: to determine the Underlying “laws” of behavior. External Validity – the extent to which the results of an experiment can be applied to and across different persons, settings, and times. (ie 14) 37

Population Validity – ability to generalize results from the sample to a larger population.

Population Validity – ability to generalize results from the sample to a larger population. Random Selection – from a population should insure generalization to the experimentally accessible population. (ie 14) 38

Target Population - larger population to which results are generalized. (ie 14) 39

Target Population - larger population to which results are generalized. (ie 14) 39

ALL PERSONS WITH DEPRESSION Persons Diagnosed Willing Control Studied (ie 14) 40

ALL PERSONS WITH DEPRESSION Persons Diagnosed Willing Control Studied (ie 14) 40

Ecological Validity: extent to which results can be generalized across settings or environmental conditions.

Ecological Validity: extent to which results can be generalized across settings or environmental conditions. (ie 14) 41

Temporal Validity – generalizability across time. Seasonal Variation - fixed-time variation - variable time

Temporal Validity – generalizability across time. Seasonal Variation - fixed-time variation - variable time variation Cyclical Variation Personological Variation e. g. , are you planning a mid-life crisis? (ie 14) 42

Treatment Variation Validity - Degree the results of a study can be generalized across

Treatment Variation Validity - Degree the results of a study can be generalized across variations in the treatment Outcome validity - Degree to which the results of a study generalize across related dependant variables. (ie 14) 43

Relationship between Internal & External Validity To obtain high Internal Validity we often create

Relationship between Internal & External Validity To obtain high Internal Validity we often create very artificial study settings. To establish external validity we often need to determine if the finding applies across setting, locations, cultures etc. (ie 14) 44

(ie 14) 45

(ie 14) 45