Data Gathering Procedures Two dimensions on which we

Data Gathering Procedures • Two dimensions on which we can distinguish data gathering procedures § The degree of contact between the researcher and the participant § The extent to which participants are given an opportunity to respond

Data Gathering Procedures • Dimensionalizing direct contact / response-gathering procedures Direct Contact Direct Response • Attitude scales • Questionnaires • Interviews Direct Contact No Response • Observational techniques No Contact No Response • Archival methods • Erosion/Accretion methods • Content analysis

Direct Contact / Direct Response • Different methods of interaction § Whether or not measure was disguised in its intent § The degree of structure imposed on participants’ responses

Direct Contact / Direct Response • Cross-tabulation of interaction types between researcher and participant Disguised More No • Information tests • Estimating group norms Less Structured Yes • Attitude scales • Thematic • Open-ended Apperception Test interviews and questionnaires • Rorschach

Direct Contact / Direct Response • The status of disguised measures § Peak of popularity was in 1950 s and 1960 s u A period when social sciences were obsessed with negative attitudes and behaviors like prejudice, obedience, conformity § Techniques emphasized control and structure, couple with deception § Techniques were probably oversold by researchers of the day § Less useful and predictive than nondisguised measures u “If you want to know something about someone, ask them; they might just tell you” (Kelly, 1955)

Direct Contact / Direct Response • Non-Disguised measures § Questionnaires and interviews § Means of questionnaire administration u Self-administered u Group-administered u Mail out § Means of interview administration u In-person interview u Focus-group interview u Telephone interview § Degree of structure u Open-ended questions u Close-ended/forced choice questions

Direct Contact / Direct Response • Advantages and disadvantages to interviews and questionnaires Advantages Interviews • More economical. Easier to give out 100 questionnaires than do 100 interviews • Data more immediate • People sometimes feel greater anonymity • Higher response rates (7080%) • In-depth probing • Don’t require literacy • Maximize seriousness of responses • Best for complex protocols Disadvantages Questionnaires • Return rates not always great (10 – 40%) • Lack of depth in probing of responses; you can’t tell if people misinterpret • Restricted utility because people must read/write • Time-consuming and expensive • Anonymity an issue: you need well-trained interviewers to build rapport • More reactive • Verbatim response? Paraphrase? Tape?

Direct Contact / Direct Response • Rating and Likert Scales § Rating scale: All things considered, how satisfied or dissatisfied are you with your life as a whole right now? 1 2 Completely Dissatisfied 3 4 5 6 7 Completely Satisfied § Likert (Likert-type) Scale: All things considered, I believe that the death penalty should be reinstated ___ Strongly disagree ___ Moderately disagree ___ Slightly disagree ___ Neutral ___ Slightly agree ___ Moderately agree ___ Strongly agree

No Contact / No Response • Unobtrusive measures § Physical trace measures u Erosion and accretion u Selective deposit and selective survival § Archival measures u Allow for longitudinal analysis u Archival sources actually exist u Lower cost u Less influenced by reactivity u Selective deposit and survival

Direct Contact / No Response • Observational techniques § What is an observation? u “The selection, provocation, recording, and encoding of that set of behaviors and organisms in situ which is consistent with empirical aims” (Weick, 1968)

Direct Contact / No Response • Methods of observational coding § What, who, and when to observer u What – Behavior sampling u Who – Subject sampling u When – Time sampling

Direct Contact / No Response • Methods of observational coding § Behavior sampling u Composite behavior u Syndrome u Simultaneous behaviors § Subject sampling

Direct Contact / No Response • Methods of observational coding § Time sampling u Why is time important? Behavior is expressed as a sequence of events Observational data is gathered over time Repeated observations are necessary The need to be able to differentiate individual differences

Direct Contact / No Response • Time sampling techniques § Semi-continuous sampling u Partial-interval sampling Divide observation period into equal time intervals Score behavior interval by interval 1 for occurrence, 0 for nonoccurrence Overestimates prevalence u Whole-interval sampling 0 for non-occurrence over entire interval; 1 for any occurrence Underestimates prevalence

Direct Contact / No Response • Time sampling techniques § Discrete sampling u Ad libitum sampling Record behaviors on a nonsystematic, informal basis Susceptible to observer bias Some behaviors attract attention u Momentary sampling Observe behavior at specific, predetermined point in time 1 if behavior occurred at that time, 0 if it did not occur Unbiased estimate of prevalence

Reliability of Observational Coding • Interobserver agreement indices § Smaller / Larger Index u Divide smaller of two values by larger of two values Observer 1: 9 vocalizations Observer 2: 10 vocalizations u Smaller/Larger Index = 9/10, 90% u Problem: Observer 1: 0 1 0 1 Observer 2: 1 0 1 0 S/L = 3/3 = 100% No individual observation agrees

Reliability of Observational Coding • Interobserver agreement indices § Percent agreement u Number of agreements divided by the number of agreements plus the number of disagreements Observer 1: 0 1 1 1 0 1 Observer 2: 0 1 1 0 0 1 u # of agree = 7, disagree = 3 u % agreement = 7 / (7+3) = 70% u Problem – inflation by chance

Reliability of Observational Coding • Interobserver agreement indices § Cohen’s Kappa u Discounts expected chance agreements u (Po – Pc) / (1 – Pc) Po = proportion agreement Pc = proportion expected by chance

Reliability of Observational Coding Percent agreement & Cohen’s Kappa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 S S S F F F R S S S F R R R S R F R S S S R F F F R R S S S F F R R S F F F S: Success in barrier crossing F: Failure in barrier crossing R: Refusal in barrier crossing Percent agreement: # agree / (# agree + # disagree) Example: # agree: 7 (S) + 4 (F) + 3 (R) = 14 # disagree: 0( S) + 2 (F) + 3 (R) = 6 Percent agreement: 14 / (14 + 6) = 70%

Reliability of Observational Coding Percent agreement & Cohen’s Kappa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 S S S F F F R S S S F R R R S R F R S S S R F F F R R S S S F F R R S F F F S: Success in barrier crossing F: Failure in barrier crossing R: Refusal in barrier crossing Percent agreement: 70% Cohen’s Kappa: (Po – Pc)/(1 – Pc) Po: Proportion agreement observed Pc: Proportion agreement expected by chance

Reliability of Observational Coding Percent agreement & Cohen’s Kappa 1 2 3 4 5 6 7 8 9 1 1 1 1 1 2 0 1 2 3 4 5 6 7 8 9 0 S S S F F F R S S S F R R R S R F R S S S R F F F R R S S S F F R R S F F F Percent agreement: 70% Cohen’s Kappa: (Po – Pc)/(1 – Pc) Observer 2 Success Observer 1 Failure Refusal Total 7 Failure 6 Refusal 7 8 5 20 Total 7 Po: (7 + 4 + 3) / 20 =. 70 Pc: ((7*7) + (6*8) + (7*5) / (20*20) = (49+4*+35)/400 =. 33 Kappa: (. 70 -. 33) / (1 -. 33) =. 5522