The measurement effect in PC smartphone and tablet

The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association Conference 13 -17 July, 2015 Reykjavik

Background and motivation Limiting surveys to the PC mode affects the size and representativeness of the sample n PC, smartphones, tablets differ in: n § § n size of the screen input method speed of connection processing power Different effect on the measurement error

Background and motivation n Smartphone Ø more break-offs longer completion time More straightlining Ø Ø § Tablet Ø Less primacy effect Ø Less straightlining Ø Less or comparable completion time Other indicators Ø Ø question order effect primacy effects open-ended answers response distribution Mixed results … but non-optimized mobile design affects completion rates, satisfaction with the survey

Background and motivation n PC and mobile response distributions are equally affected by response formats Drop-boxes –preference for first options Grids – preference for visible options, straightlining - no conclusive evidence which format is better for mobiles

Research Question How mode of administration and response format affect survey responses n Indicators of measurement error: § § overall satisfaction rates straightlining response distribution length of open-end questions

Survey and Questionnaire design n n n National Satisfaction Survey 2014 Administered to final year higher education students in UK 22 core questions using a 5 -point Likert scale 2 open-end questions Mixed-mode: self-selected mail, phone, web 5 response formats – randomly allocated

Sample composition Radio-button PC N=5529 Web Survey Sample N=9276 Radio-button Smartphone N=3196 Tablet N=551 Drop-box (+) Drop-box (-) Drop-box (0) Drop-box ()

Screenshots: response design • Once clicked a list with options appears on a separate screen. • Responses always in the same order

Screenshots: response design • Requires vertical scrolling. • All questions are visible on the screen. • Requires horizontal scrolling. • Portrait or landscape viewing.

Measurement Effect between PC smartphone tablet responses Comparisons made across all three modes, but within one radio-button response format 10

Measurement Effect between PC smartphone tablet MANOVA to test for differences F=2. 3, p<. 05 F p PC smartphone tablet Straightlining 3. 9 <. 05 6. 9% 10. 3% 6. 4% Modal responses <1 >. 05 Mean values 2. 7 . 07 4. 1 4 Positive feedback 1. 3 >. 05 195 180 191 Negative feedback 1 >. 05 257 240 243 ‘Mostly Agree’ ‘Definitely Agree’ & ‘Mostly Agree’

ME between PC, smartphone, tablet summary n Smartphone straightlining significantly higher n Tablet straightlining rate the lowest n Signs of the visibility effect for smartphone n Other quality indicators comparable between all three modes

Measurement Error between different response designs presented on a smartphone and a tablet Comparisons made across five survey response formats within smartphone and tablet modes 13

n MANOVA test of differences F=2. 7, p<. 05 F p device 2. 7 <. 05 format 1. 7 <. 05 interaction 1. 3 . 18 § Both device and format affected data quality

Responses in Drop-box with a positive initial option Smartphones 70, 00% 60, 00% § higher selection of initially suggested response. § less moderately positive responses § 11. 1% straightliners § Mean 4. 3 50, 00% 40, 00% 30, 00% Tablets 20, 00% 10, 00% Def. Agree Mostly Agree Neither smartphone Mostly Def. Disagree tablet § Similar selection of extreme and moderately positive responses § More negative responses § 5. 6% straightliners § Mean 4. 1

Responses in Drop-box with a negative initial option 50, 00% Smartphones 45, 00% § § 40, 00% 35, 00% 30, 00% 25, 00% More extreme negative responses Less positive responses 8. 4% straightliners Mean 4 20, 00% Tablets 15, 00% 10, 00% 5, 00% 0, 00% Def. Agree Mostly Agree Neither smartphone Mostly Disagree tablet Def. Disagree § More positive responses § Extremely low ‘Definitely Disagree’ frequency § 2. 5% straightliners § Mean 4. 1

Responses in Drop-box with a middle initial option 45, 00% Smartphone and Tablet 40, 00% Comparable results 35, 00% § Weak evidence of selecting the middle option § Most frequent ‘Mostly Agree/Disagree’ selection across formats § 6. 5% and 6. 1% of straightliners § Mean 4 an 3. 9 30, 00% 25, 00% 20, 00% 15, 00% 10, 00% 5, 00% 0, 00% Def. Agree Mostly Agree Neither Mostly Disagree smartphone tablet Def. Disagree

ME indicators summary n Smartphone – answers affected by response formats. Ø Initially suggested response is selected more often Tablet – no strong effect of response formats. n No significant differences in the length of open answers between formats. n

Potential Limitations n n n No reverse-coding Similar question wording Not counterbalanced question or response order Instructions universal for each response format Self selected device condition Survey sample: highly educated, IT literate, similar age, highly motivated

Thank You vkolba@essex. ac. uk