What is meant by mode effect on measurement

What is meant by mode effect on measurement? A research study to identify causes of mode effects Gerry Nicolaas

Background • Increasing use of mixed modes 4 Falling response rates 4 Rising costs of data collection • Risk of reduced data comparability 4 Coverage error 4 Non-response error 4 Measurement error • Need for practical advice to inform decisions about when to mix modes and how

Mixed Modes and Measurement Error • Funded under the ESRC Survey Design and Measurement Initiative • 3 -year contract starting 1 Oct 2007 • Collaboration between Nat. Cen, ISER and independent survey methods consultant

Research Team • National Centre for Social Research (Nat. Cen) 4 Gerry Nicolaas 4 Steven Hope 4 David Hussey • Institute for Social & Economic Research (ISER) 4 Peter Lynn 4 Annette Jäckle 4 Alita Nandi 4 Nayantara Dutt • Independent Survey Methods Consultant 4 Pam Campanelli

Main Objective • Practical advice on how to improve portability of questions across modes 4 Which mode combinations are likely to produce comparable responses? 4 Which types of questions are more susceptible to mode effects?

Research Design • A literature review & framework of mixed modes 4 develop a conceptual framework 4 identify gaps in evidence base and formulate hypotheses to address gaps • Quantitative data analysis 4 test hypotheses using existing datasets and new experimental data • Cognitive interviewing 4 explore how respondents process

Mode is defined by: • Interviewer presence (face-to-face, phone, none) • Oral and/or visual transmission of information 4 Question delivery 4 Delivery of response options 4 Recording of response

Face-to-Face Interview Question Response Options Response Description Aural Oral FTF (no card) Aural Visual Oral FTF (card) Visual (CAI) Written (CAI) CASI Visual (paper) Written (paper) Aural (CAI) Written (CAI) SAQ in-int’w Aural (CAI) ACASI Visual (CAI) Written (CAI) ACASI

Telephone interview Question Response Options Response Description Aural Oral Telephone Aural Written (CAI) TDE Aural Visual Oral Aural (rec) Oral (CAI) Phone with showcards IVR

Self-completion Question Response Options Response Description Visual (paper) Written (paper) Visual (CAI) Written (CAI) SAQ (e. g. mail) Web / email Aur & Vis (CAI) A-Web Aur & Vis (CAI) Written (CAI)

Causality • How, when and why to mix/choose modes • Need to better understand causal mechanisms • Development and testing of behavioural theory

Privacy/legitimacy I-R interaction ias b y Social norms: bilit a r i s de l Willingness to disclose? a i oc Interviewer presence S Comprehension Retrieval Context information: Influence on processing? Sequential/ Simultaneous Qs Control Qaire Judgement Response g Depth of cognitive processing: n i ic f Sufficient Effort? s i t a S Add. explanations I characteristics Time pressure R distraction R motivation Interviewer presence Cognitive demands Aural/visual

Hypotheses • Short versus long response lists 4 Effects of interviewer presence on satisficing • Agree-Disagree scales 4 Is acquiescence caused by satisficing, cognitive ability, social desirability? • Ranking versus Rating 4 Effects of interviewer presence • Fully-labelled versus End-labelled scales 4 Effects of visual stimulus and interviewer presence • Showcards versus No Showcards (face-to-face interview) 4 Effects of visual stimulus • “Branched” versus “Non-branched” questions 4 Effects of item design (no mode differences expected) • “Yes/No” versus “Code all that apply” 4 Effects of item design (no mode differences expected)

Question selection Questions were designed to vary by: • Task difficulty • Sensitivity • Question type 4 Satisfaction 4 Other attitudinal 4 Behavioural 4 Other factual

Mixed Modes experiment • Follow-up surveys to Nat. Cen Omnibus (& BHPS) 4 Face-to-face, telephone and web comparisons 4 Experimental design with random allocation • Limitation 4 Restricted to respondents with web access

Acquiescence (1) • Hypothesis 4 Acquiescence is a form of satisficing 4 More satisficing in web than f 2 f & tel (no interviewer to motivate, explain, probe, etc) • 12 Agree/Disagree questions 45 -point agree/disagree scale 4 Use of opposite statements

Acquiescence (2) • Initial results from the experiment 4 More acquiescence in f 2 f & tel compared to web • Results from the cognitive interviews 4 Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence 4 Justifiable explanations given for other 21 cases

Acquiescence (3) • Example of justifiable agreement with opposite statements 4 N 36: Compared to other neighbourhoods, this neighbourhood has more properties that are in a poor statement of repair. 4 N 38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept. 4 Respondent: In this village, … it’s like half and half…”

Acquiescence (4) • Interim conclusions 4 Use of opposite statements to detect acquiescence bias brought into question 4 Why higher rate of acquiescence in f 2 f & tel compared to web?

Other forms of satisficing (1) • Hypotheses: 4 More satisficing in web than f 2 f & tel (no interviewer to motivate, explain, probe, etc) 4 More satisficing in tel than f 2 f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc) • Indicators of satisficing 4 Primacy effects in visual modes, recency effects in aural modes, middle

Other forms of satisficing (2) • Initial results from the experiment 4 Primacy & recency effects: inconsistent patterns 4 Item non-response: no mode differences 4 Middle category effects: web respondents more likely to select middle categories than f 2 f & tel resps (also for agree/disagree scales – see acquiescence results) • Results from cognitive interviews 4 Results suggest more satisficing in web &

Mode effect or question design effect? • Example: 4“Code all that apply” frequently used in f 2 f interviews and self-completion surveys 4 Alternative format for tel interviews tends to be a series of Y/N questions • Hypothesis 4 No mode effect 4 i. e. No differences if series of Y/N questions used across all modes

Mode effect or question design effect? • Initial results from the experiment 4 F 2 F & tel resps more likely than web resps to say “Yes” in series of Y/N questions 4 Similar mode effect not found for f 2 f and web respondents in “code all that apply” format (nb this format not used in tel mode) • Results from cognitive interviews 4 Questions raised about validity of “Yes”

What next? • Continue with analysis of experimental data • Papers 4 Causes of mode effects on survey measurement 4 The role of the interviewer in producing mode effects 4 The role of visual/aural stimuli in producing mode effects 4 The role of question format in producing ‘mode’ effects 4 Using cognitive interviews to explore mode

For more details on this project, contact: Gerry. Nicolaas@natcen. ac. uk