Chapter 5 Reliability and Validity Power Point presentation

Overview Measuring Variables l Choosing a Behavior to Measure l Overview of Types of Measurement Errors Manipulating Variables l Validity – Threats to – Establishing l – Bias – Random error l l Reliability Validity *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition; Types of manipulations

Two Types of Measurement Error l l Bias Random error *Power. Point presentation to

Three “Places” Measurement Error Can Occur l l l Observer/Scorer Participant Person administering the measure *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Two Types of Observer Error l l Observer bias (Scorer bias) Random observer error

Minimizing Observer Errors l l Why it is more important to reduce observer bias than random error Techniques for reducing observer bias* *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Techniques for Reducing Observer Bias l l Eliminating human observer errors by eliminating the human observer Limiting human observer errors by limiting the human observer’s role Reducing observer bias by making observers “blind” Conclusions about reducing observer bias *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Reducing Random Observer Error l Most of the techniques that reduce observer bias reduce random observer error *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Errors in Administering the Measure l Types – Experimenter (researcher) bias – Random error l Solutions – Blind technique to reduce bias – Standardization to reduce both bias and random error *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Errors Due to the Participant l l Bias due to the participant (Subject bias) Random error due to the participant *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Subject (Participant) Bias l l Obeying demand characteristics Social desirability bias *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Conclusions about Reducing Subject Biases l l Blind techniques can reduce demand characteristics Making participants anonymous can reduce social desirability bias *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Summary of Types of Measurement Error l l Try to reduce all forms of measurement error Really focus on reducing bias *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Reliability: The (Relative) Absence of Random Error l l The importance of being reliable: Reliability as a prerequisite to validity Using test-retest reliability to assess overall reliability: To what degree is a measure “random error free”? *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Identifying (and Then Dealing with) the Main Source of a Measure’s Reliability Problems l l l Are observers to blame for low test-retest reliability? : Assessing observer reliability Non-observer sources of random error Using internal consistency measures to estimate random error due to participants *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Internal Consistency: Test Questions Should Agree with Each Other l Random error due to participants may cause low internal consistency *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Two Solutions to Problems Caused by Random Participant Error l l Add questions to let random participant error balance out Ask better questions to reduce random participant error *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Measuring Internal Consistency l l Average inter-item correlations as indexes of internal consistency Split-half coefficients as indexes of internal consistency Additional indexes of internal consistency Conclusions about internal consistency’s relationship to reliability *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Conclusions About Reliability l l l Reliability is a prerequisite for validity If test-retest reliability is low, try to find out where reliability problem is and fix it. Reliability does not guarantee validity *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Beyond Reliability: Establishing Construct Validity l l l Content Validity Internal Consistency Convergent Validity: Getting evidence that you are measuring the right construct *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley l 6 th edition; Discriminant Validity: Showing that you are not measuring the wrong construct

Manipulating Variables l l Common threats to a manipulation’s validity Evidence used to argue for a manipulation’s construct validity Tradeoffs among three common types of manipulations Conclusions *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Common Threats to a Manipulation’s Validity l l l Random error Experimenter bias Subject biases *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Evidence Used to Argue for a Manipulation’s Construct Validity l l Consistency with theory Manipulation checks *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Tradeoffs Among Three Common Types of Manipulations l l l Instructional manipulations Environmental manipulations Manipulations involving stooges *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;

Concluding Remarks Operational definitions should l Be consistent with dictionary/theory definitions l Be standardized to reduce bias and random error l Have evidence to support their validity *Power. Point presentation to accompany Research Design Explained © 2007 Mark Mitchell & Janina Jolley 6 th edition;