Research Methods An Introduction Priya Raghubir Haas School

Research Methods: An Introduction Priya Raghubir Haas School of Business April 13, 2004 Berkeley Consulting

Validity, Reliability, Representativeness n n Validity: Asking the right question Reliability: Receiving the right response n n Reducing Response Error Representativeness: Generalizing results. n n Sample selection and sample size Reducing Non-response Error April 13, 2004 Berkeley Consulting 2

Reliability n n Similar results by comparable measures of the same construct Two ways to measure reliability n n n Test-Retest: : stability over time Split-Half: consistency across ratings within a multiple-item scale Reliability is easier to determine than Validity n n n If a measure is valid, then it is reliable If it is not reliable, it can not be valid If it is reliable, it may or may not be valid April 13, 2004 Berkeley Consulting 3

Introducing New Coke: Research Conducted n 2000 interviews in 10 major markets: n n n Focus Groups: Mixed; loyal drinkers unfavorable. Blind Taste Tests: n = 30, 000 -40, 000 n n Control for brand names; Storyboards and Mock Advertisements: Try? Switch? Be Upset? New vs. Old Coke: 55 vs. 45; vs. Pepsi: 54 -46 Generalizability: n=191, 000 in 13 cities; Blind taste tests of 4 new vs. old / Pepsi. April 13, 2004 Berkeley Consulting 4

New Coke: Key Research Design Takeaways n Predictive Validity n n Construct Validity: n n Did not research impact of the Loss of Coke Buy brand vs. prefer taste? Reliability: n Discriminate/ Prefer April 13, 2004 Berkeley Consulting 5

New Coke: Key Research Implementation Takeaways n Sample Size: n n Data Analysis: n n Ignoring focus group dissent Conclusions: n n Effect Size vs. Significance 39% did not prefer new taste in branded taste test; 45% in blind taste test! But then, there is Hindsight Bias … April 13, 2004 Berkeley Consulting 6

Biases In Answering Survey Questions Basic tenet: Ask a question, and you’ll get a response! Translation: Garbage in, garbage out! April 13, 2004 Berkeley Consulting 7

Examples of Questions from Consumer Surveys (The Wall Street Journal, Nov. 14, 1991) n n When Levi Strauss & Co. asked students which clothes would be most popular this year, 90% said Levi’s 501 jeans. (They were the only jeans on the list. ) A survey for Black Flag said: “A roach disk… poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches? ” Not surprisingly, 79% said effective! April 13, 2004 Berkeley Consulting 8

Examples… (continued from previous slide) n A Gallup poll sponsored by the disposablediaper industry asked: “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers? ” (84% said no. ) April 13, 2004 Berkeley Consulting 9

Sources of Error in Surveys n Non-response bias due to refusals n n Inaccurate responses n n n fear of consequences of participation, represents an invasion of privacy, anxiety about the subject inability to respond: ignorance, memory problems, problems formulating an answer unwilling to respond accurately: concerns about invasion of privacy, time pressure/fatigue, prestige seeking/social responsibility, response style/bias Error causes by interviewers n n providing clues to answers inadequate questioning or probing April 13, 2004 Berkeley Consulting 10

Reducing Response Errors n How do respondents answer survey questions? n n n Guess Randomly choose an option Use Information from previous response: order effects Social Desirability Bias Memory (Forgetting) Use information available in the survey context to generate response April 13, 2004 Berkeley Consulting 11

Improving Reliability: Increase Respondent Motivation n n Purpose of the Research, its legitimacy Incentives to respond accurately n n Monetary: $1; Affective: guilt ; Informative: Use research will be put to; Conative: sweepstakes etc. Disentangle collecting data from selling Personal questions at end Include “Other ____” for own perspective April 13, 2004 Berkeley Consulting 12

Improving Reliability: Reduce Respondent Effort n Short questionnaire n n n Provide time it takes to respond. Order of questions should aid memory Provide mnemonic cues: decomposition Most important measure should come first Use closed-scaled responses when possible April 13, 2004 Berkeley Consulting 13

Improving Reliability: Multiple Measures n Ask a question in different ways and assess correlation of responses (Cronbach’s ) n n n Use Multiple scales: open- and closed-ended. n n Max Price willing to pay/ price evaluation/ quantity @ different prices. Counterbalance order of key measures: n n “Attitude”: “bad-good” “poor-excellent” “Intention”: try, buy, recommend, repurchase General-Specific versus Specific-General Measure respondent motivation and interest April 13, 2004 Berkeley Consulting 14

Improving Reliability: Use Appropriate Method n Does the respondents know the response? n n Is the respondent willing to share the response? n n n Is it “accessible” to them? Can they “explicate” it? Can they “recall” it or merely “recognize” it? Counter-biasing methods; Anonymity and confidentiality If not, supplement descriptive data with n Observation studies Experimental studies n April 13, 2004 Berkeley Consulting 15

The New York Times, 7/8/94 April 13, 2004 Berkeley Consulting 16

How to ask a sensitive question (Barton, POQ 1958) n The casual approach: "Do you happen to have killed your partner? ” n The numbered card approach: "Would you please read off the number on this card which corresponds to what became of your partner? 1 2 3 April 13, 2004 Natural death I killed her/ him Other" Berkeley Consulting 17

How to ask a sensitive question (Barton, POQ 1958; continued) n The everybody approach: (also called counterbiasing method) "As you know, many people have been killing their partners these days. Do you happen to have killed yours? " n The other people approach: "Do you know any people who have murdered their partners? How about yourself? " April 13, 2004 Berkeley Consulting 18

Sensitive questions (continued) n n Personal Information - Build rapport, legitimize Embarrassing Information (downward bias) ° ° ° n counterbiasing statement: “Recent studies indicate that a man often uses his wife’s hair spray. Do you use your wife’s hair spray? ” projective techniques randomized response techniques Prestige, Normative Information (upward bias) April 13, 2004 Berkeley Consulting 19

Asking Sensitive Questions: Rules of Thumb n n n Hide in a group of more innocuous questions State that the behavior or attitude is not unusual Phrase in terms of others and how they might feel or react Use multiple-choice response categories Randomized response April 13, 2004 Berkeley Consulting 20

Watch Out: Garbage In, Garbage Out Behavioral Questions: Memory Biases Omission Telescoping Attitude Questions: Confusion Double-barreled, leading, ambiguous questions Sensitive Questions: “Lying” Item non-repsonse Termination Distortion April 13, 2004 Berkeley Consulting 21

Questionnaire Design: Main Take-aways n Questionnaire Design is like Advertising n n n Questionnaire Design is a Science n n The respondent is not interested in answering your questions. You need to make it worth their while, and EASY for them to give you their opinions, behaviors. Use information about memory-models and attitude models to generate your questions. Questionnaire Design is an Art n n There is no substitute for experience, so Carry out a pretest, revise and then do final draft April 13, 2004 Berkeley Consulting 22