What happens if you offer a mobile option

What happens if you offer a mobile option to your web panel? Results from a pilot study comparing a desktop and mobile device survey Vera Toepoel (Utrecht University, V. Toepoel@uu. nl) Peter Lugtig (Utrecht University/University of Essex, p. lugtig@uu. nl) 1

Vast increase in mobile phone use u According to Statistics Netherlands, 50% of all 12 to 72 year olds with Internet access do this via a mobile device in 2011 u Smartphone penetration rate explosion – In 2011 1 out of 5 people opened their email on a smartphone (source: Emerce) http: //www. emerce. nl/research/mobiel-emails-lezen-2011 -verdubbeld 2

Smartphone as research tool? u Online probability based panel of Market Response – RDD, Internet population – Panel members ask if they can complete the survey on their mobile phone – Young panel members (age 18 -34) have the lowest response probabilities in the panel and are continuously underrepresented in the panel (reasons for non response: don’t feel like it, no time) – Current software not capable of using mobile friendly design u Confirmit software offers ‘responsive design’ (survey automatically optimized for the device chosen) 3

Research Design Two parallel surveys u June 2012 two surveys with similar sampling frame – (NL, 18+, Internet users, mobile registration): – Standard QSL (current online method) – Confirmit with response design (respondent could chose prefered device) » Soft check for mobile completion Invitation: QSL Confirmit 4

Research Questions What happens if you offer a mobile option to your web panel? u Amount of people using mobile – Substantial? u Socio-demographic characteristics – Coverage bias: age, edu, gender, urbanization, hh size+composition u u Response (break offs and item non-response) GPS Evaluation Response quality (measurement error) – Questionnaire already adapted to mobile (no experiment) – Open-ended questions, check-all-that-apply, primacy 5

Research Questions Amount of people using mobile u 57% used mobile phone throughout the survey (N=443) – 252 used mobile (57%) – 33 people switched from desktop to mobile during the survey (7%, 12% of mobile starters) – 4 people switched from desktop to mobile (1%) – 154 people used desktop (35%) – Note: we define mobile as smart phone only. Additional analyses with mobile=all with touchscreen (incl. ipads etc. ) produces similar results. 6

Research Questions Socio-demographics u Age, hh size and hh composition (kids) significant predictors of mobile use – Older people – Larger households – No kids u No effect of: urbanization, gender, social class, education, household position, family income 7

Hard-to-reach-group Young: mobile increases response probability 8

Research Questions Response u Break offs limited – – u Short questionnaire 4 out of 285 break off on mobile 3 out of 153 on desktop No evidence that mobile results in higher break offs Item non-response – 7% over 3 questions – No significant difference mobile versus desktop – No significant difference in non-substantive answers (don’t know) 9

Research Questions time to complete 10

Research Questions Day questionnaire completed 11

Research Questions GPS Mobile Desktop Total GPS ok 38% 8% 25% N 253 190 443 % Home Work On the road outside abroad Mobile 67 14 11 6 2 N=110 12

Research Questions Evaluation u u u % Mobile Desktop Total confirmit survey qsl Positive (8 -10) 42 44 43 38 Neutral (6 -7) 50 47 49 55 Negative (1 -5) 8 9 8 7 n(mobile)=281, n(desktop)=151, N(QSL)=498 Chi-square test across mobile and desktop not significant Chi-square test across all three groups not significant 13

Evaluation Higher evaluation young people 18 -24 7. 7 25 -34 7. 1 35 -44 7. 5 45 -54 7. 1 55 -64 6. 9 65+ 6. 4 7. 2 14

Research Questions Response Quality: open-ended questions % Mobile Desktop No answer 35% 33% Total (Confirmit) 33% 0 -50 characters 52% 49% 51 -100 charcters 100 + characters 10% 16% 13% 3% Notes: N(mobile)=285, N(desktop)=154 No significant difference 15

Research Questions Response Quality: check-all-that-apply % 0 1 2 3 4 5 6 7 8 Mean number of shops visited Mobile 24. 2 9. 1 15. 4 14. 7 14. 4 8. 8 7. 0 4. 6 1. 8 2. 7 Desktop 21. 4 9. 7 18. 2 12. 3 15. 6 8. 4 6. 5 4. 5 3. 2 2. 8 Total (Confirmit) 23. 2 9. 3 16. 4 13. 9 14. 8 8. 7 6. 8 4. 6 2. 3 2. 8 Number of shops visited in past year Notes: N(mobile)=285, N(desktop)=154 No significant differences 16

Research Questions Response Quality: primacy in radio buttons % One a week or more often About once a month About every 3 months About once every 6 month Once a year o less often Don’t know Mobile 3. 9 Desktop 0. 6 Total Confirmit 2. 7 23. 9 27. 7 25. 2 37. 2 32. 1 35. 4 21. 4 26. 4 23. 2 10. 2 9. 4 9. 9 3. 5 3. 8 3. 6 Notes: N(mobile)=285, N(desktop)=154 Frequency of buying clothes in a shop nearby No sigificant difference 17

Feedback mobile option Perfect format, excellent speed! Super! The scrollbar was not working properly. Did not like it. it is easier on desktop(evaluation 3: woman, 58) I am glad you figured out that desktop pcs are getting used less often and that questionnaires should be made mobile friendly. I don’t need to get my laptop (evaluation 8: woman, 29) Looovvedddd it(evaluation 9: woman, 18) I like this very much. I do not need to get my laptop (evaluation 8: man, 20) Very good!!! I will definetely get higher participation rates if this will be implemented (evaluation 9: man, 32) Good. But it should not take me too long, otherwise I’ll break off(evaluation 8: man, 51) Good! I often complete surveys on my mobile phone but this is the first one that reads well, perfect format, excellent speed! Super! (evaluation 9: woman 24) Handy! Better then the desktop since I tend to forget the survey invitations (evaluation 8: woman, 38) Don’t like it. I have a pc which I use for all my survey activities. Next time I won’t be doing the survey(evaluation 1: man, 71) 18

Conclusion u u u u Considerable amount (57%) uses mobile No effect on non-response No effect on response quality Similar evaluation GPS gives additional insights No reason to believe that mixed-device is a problem Able to attract hard-to-reach group such as young people 19

Discussion u What about difficult questionnaires? – Long – grids u u u What about privacy? How to offer additional information to researchers? Coverage bias? Thank you! 20