Experiments with Loss Framing Verbiage in Survey Introductions

Experiments with “Loss Framing” Verbiage in Survey Introductions to Raise Response Rates Paul J. Lavrakas, Benjamin Phillips, Shane Compton, and Darren Pennay WAPOR 2019, Toronto, Canada A subsidiary of:

The Problem: Reducing Nonresponse • Survey researchers strive to reduce nonresponse primarily to: • Raise response rates – quality metric • Ostensibly gain a more representative final sample • Ostensibly reduce possible nonresponse bias • Reduce total survey costs www. srcentre. com. au 2

Loss Framing • Tourangeau and Ye (2009) reported an experiment that showed a marked gain (10 pp) in survey response when the respondent was told that not participating in the survey would “harm” the research/researchers compared to when the respondent was told that participating in a survey would “help” • This “loss framing” strategy for survey research followed from Kahneman and Tversky’s (1979) “Prospect Theory, ” which posits that people tend to be more sensitive to losses than to gains of equal magnitude • Others (e. g. , Sakshaug, 2014) have reported similar findings about Loss Framing in survey research • Of note, many of these experiments have been conducted using respondents that already had some experience with the researchers www. srcentre. com. au 3

Loss Framing 2 x 2 Experimental Design • Past Loss Framing experiments to raise response rates have tested as many as three conditions in the introductory request: • Mention of Help • Mention of Harm • No mention of either (Control) • In 2018, we conducted two 2 x 2 experiments about Loss Framing in Australia in 2018, in which we tested mentioning (1) Harm by itself, (2) Help by itself, (3) Both Harm and Help, and (4) a “no Help, no Harm” control condition • We also conducted a simple Harm vs. Control (No mention of Harm) experiment within a later wave of one of the studies that we used for the 2 x 2 experiments www. srcentre. com. au 4

Loss Framing 2 x 2 Experimental Design • Basic survey introductory request was an email: Dear XXX, As a current student I would like to hear about your experiences of the [course] at your institution though the XX survey. The XXX is the largest Australian study of education and part of the Quality Indicators for Learning and – the only source of national data on student experiences with higher education. To start the survey, please click on the link below: ***EXPERIMENTAL TEXT INSERTED HERE*** If you are unable to access the survey by clicking on the link, please copy and paste the link into a new browser window. Alternatively, go to www. XXX. edu. au, click ‘start survey’ and enter your unique login details as below. Username: %%Username%% Password: %%Password%% It should take approximately 10 minutes to complete and your responses will be kept confidential. More information about the research is provided at XXX including contact information if you have any queries or technical issues with the survey. Your ideas and opinions are important to future students. Thank you in advance for your time and feedback. www. srcentre. com. au 5

Loss Framing 2 x 2 Experimental Design • ONLY Help: Complete the [questionnaire] to contribute to improvements in the [course] at your institution. Because of student participation in the survey, institutions have improved student life and teaching. • ONLY Harm: If we don’t receive enough responses from students like you, we won’t be able to publish your area of study on the XXX website and future students won't know what it's like to study at your institution. • BOTH Help + Harm: Complete the [questionnaire] to contribute to improvements in the [course] at your institution Because of student participation in the survey, institutions have improved student life and teaching. If we don’t receive enough responses from students like you, we won’t be able to publish your area of study on the XXX website and future students won't know what it's like to study at your institution. • NEITHER HELP NOR HARM: None of the above verbiage www. srcentre. com. au 6

Loss Framing 2 x 2 Experimental Design • The two experiments had to be worked into on-going studies in ways that would not harm those studies, this limited what we could do with the wording of the experimental treatments • In Study 1, the experiment was carried out as part of the 3 rd reminder sent to nonresponders • In Study 2, the experiment was carried out as part of the 9 th reminder sent to nonresponders www. srcentre. com. au 7

Experiment 1 Percentage Completed Questionnaire after Reminder #3 Was HARM Mentioned? Was HELP Mentioned? NO YES ROW TOTAL NO 28. 3 28. 4 28. 3 YES 28. 2 28. 4 28. 3 COLUMN TOTAL 28. 2 28. 4 28. 3 • n = 559, 791 • ANOVA: No Significant Main Effects or an Interaction Effect www. srcentre. com. au 8

Experiment 2 Percentage Completed Questionnaire after Reminder #9 Was HARM Mentioned? Was HELP Mentioned? NO YES ROW TOTAL NO 3. 5 YES 3. 4 3. 5 COLUMN TOTAL 3. 5 • N = 68, 994 • ANOVA: No Significant Main Effects or an Interaction Effect www. srcentre. com. au 9

Experiment 3 • Respondents (n = 25, 122), who had already completed the original questionnaire, were later asked within the survey to provide information about their work supervisor to generate sample for the Employer Satisfaction Survey • Historically this questionnaire has had a very high item non-response: 10% provide information < • For the experiment, 10% of respondent pool was randomly allocated to Loss Framing due to concern about possible negative reaction • The experiment was analyzed part-way through field period • Due to success of Loss Framing verbiage over control, experimental allocation was reversed mid-field, with 90% getting the Loss Framing treatment www. srcentre. com. au 10

Experiment 3 • CONTROL VERBIAGE: For the next part of the study we would like to hear from your work supervisor about their perceptions of your institution and higher education broadly through the Employer Satisfaction Survey (ESS). Please click next to continue. • LOSS FRAMING VERBIAGE: For the next part of the study we would like to hear from your work supervisor about their perceptions of your institution and higher education broadly through the Employer Satisfaction Survey (ESS). Without your supervisor’s input, results from this survey will be less useful to policy makers. The government uses input from graduates and employers to understand how well higher education institutions are preparing graduates for the workforce. Please click next to continue. www. srcentre. com. au 11

Experiment 3 Percentage Answering the Question No Loss Framing I can provide their contact work details 8. 9 10. 3 I can provide their contact information but I wish to log out of the survey and check their details first 0. 3 0. 5 I can provide their contact information but I would like you to call me 0. 4 0. 9 88. 5 86. 1 I would like more information before I provide my supervisor’s details 1. 9 2. 1 PERCENTAGE THAT GAVE AN “I CAN” REPLY 9. 6 11. 7 I do not wish to provide my supervisor’s details Chi-square: p <. 001 • 22% increase in agreeing to provide data about supervisor www. srcentre. com. au 12

Discussion • Our experiments suggest that Loss Framing is likely to be most effective when the respondent already has some experience with what the researchers are requesting • This follows from the logic that if a respondent has made some effort to participate in a study, telling her/him that the value of the data already provided may be lowered if s/he does not continue to participate (e. g. , linking the value of one’s data in the current wave of a panel to one’s prior data in a panel) • This does not mean that Loss Framing will have no positive effects with “first time” respondents • More research is needed to better understand how best to operationalize the Loss Framing message www. srcentre. com. au 13

Thank You! www. srcentre. com. au 14