Handbook of Online Research Methods Colloquium 28 29

Handbook of Online Research Methods Colloquium 28 -29 March 2007 Overview: Online Surveys Vasja Vehovar University of Ljubljana, Slovenia http: //Web. SM. org

Introduction Structure • Computer-assisted survey information collection • Key methodological issues • Related issues • Applications

Computer-assisted survey information collection Technology and surveys 1930 s: applications of probability sampling 1960 s: expansion of telephone surveys 1970 s: computer technology appears in surveys 1980 s: computer-assisted surveying 1990 s: Internet mediated surveys

Computer-assisted survey information collection Early CASIC interviewer assisted modes • CATI – Computer-assisted telephone interviewing • CAPI – Computer-assisted personal interviewing • CASI – Computer-assisted self-interviewing

Computer-assisted survey information collection CASIC benefits • Reduced time and costs for data input • Elimination of errors during data transcription • Implementation of advanced features: Ø automatic skips and branching Ø randomization of questions and response options Ø control of answers Ø inclusion of multimedia elements…

Computer-assisted survey information collection Computerized self-administered questionnaires (CSAQ) Different modes of CSAQ: Ø disk-by-mail Ø touch-tone data entry (TDE) Ø interactive voice response (IVR) Ø e-mail surveys Ø web surveys

Computer-assisted survey information collection Online, Internet, Web, CASIC, CSAQ, CADAC CASIC Online surveys (e. g. CATI, CAPI, CASI, etc. ) Internet surveys Web surveys

Computer-assisted survey information collection Online, Internet, Web, CASIC, CSAQ, CADAC CSAQ CASIC Online surveys (e. g. CATI, CAPI, CASI, etc. ) Internet surveys Web surveys

Computer-assisted survey information collection Online, Internet, Web, CASIC, CSAQ, CADAC CSAQ CASIC Online surveys (e. g. CATI, CAPI, CASI, etc. ) Internet surveys Web surveys CADAC

Computer-assisted survey information collection Interviewer-less and paper-less surveys Interviewer involvement Survey mode Paper and pencil CASIC Interviewer presence Paper and pencil (face-toface) interviewing (PAPI) CAPI, CASI, Audio/Video CASI Remote interviewer Paper assisted telephone interviewing (PATI) CATI, CAVI (computer assisted video interviewing) No interviewer Self-administered paper questionnaires (mail questionnaires) CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer, . . . )

Computer-assisted survey information collection Interviewer-less and paper-less data collection Interviewer involvement Survey mode Paper and pencil CASIC Interviewer presence Paper and pencil (face-toface) interviewing (PAPI) CAPI, CASI, Audio/Video CASI Remote interviewer Paper assisted telephone interviewing (PATI) CATI, CAVI (computer assisted video interviewing) No interviewer Self-administered paper questionnaires (mail questionnaires) CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer, . . . )

Computer-assisted survey information collection CSAQ input-output technology Asking question (INPUT) Recording of responses (OUTPUT) Manual recording Automatic voice recognition Written questions Standard CSAQ (PC, TV, PDA, Mobile, . . . ) Visual CSAQ with AVR Audio questions Video CSAQ, Audio CSAQ, TDE IVR, Video IVR, TTS with AVR, Virtual interview

Computer-assisted survey information collection CSAQ input-output technology Asking question (INPUT) Recording of responses (OUTPUT) Manual recording Automatic voice recognition Written questions Standard CSAQ (PC, TV, PDA, Mobile, . . . ) Visual CSAQ with AVR Audio questions Video CSAQ, Audio CSAQ, TDE IVR, Video IVR, TTS with AVR, Virtual interview

Computer-assisted survey information collection Technological aspects • Developments of the Internet and related technologies • Importance of broadband Internet access • Possibilities of distribution across various platforms and devices • Fast and easy implementation using specialized software tools

Key methodological issues 1. Probability vs. non-probability sampling • Two major problems of probability samples in Internet surveys: Ø non-coverage Ø sampling frame problems • Image of Internet surveys as inherently non-probability ones

Key methodological issues The spurious link Issue of a statistical inference: probability vs. non-probability samples Information-communication technologies Internet survey Management of the survey process Mode of survey data collection

Key methodological issues 2. Sampling and invitation Sampling type Solicitation Type of invitation Personal invitation General invitation Probability samples Samples from closed population with email, Pre-recruited with mail, telephone, f 2 f Non-probability samples Self-recruited Internet panels, Opt-in lists of e-mail adresses Web intercept surveys General banner-ad invitation

Key methodological issues 3. Non-response • Generally low and variable response rates to web surveys • Complexity of the response process • Measures to increase response rates: Ø incentives (e. g. using Pay. Pal) Ø multiple follow-up contacts Ø other motivations

Key methodological issues Involvement process Target population Complete respondents Operational population Partial respondents Frame population Starting units Sampled units Solicitation Clickers Absorbed units Informed units Aware units

Key methodological issues Participation in web surveys Respondent’s characteristics No researcher’s control Social environment Researcherrespondent interaction Survey design Technological environment Partially under researcher’s control

Key methodological issues 4. Questionnaire design • Question types and visual elements • Advanced features of computerized questionnaires • Inclusion of multimedia • Problems and considerations: Ø standardization Ø impact on respondent’s answers Ø technical problems measurement error

Key methodological issues 5. Post-survey adjustments • Commonly utilized for correcting: Ø deviations from probability selections Ø non-coverage and non-response problems • Importance for non-probability Internet surveys • Promising approaches: Ø calibration methods Ø propensity score weighting Ø modelling and causal analysis

Related issues 1. Costs, errors and management • The managerial process of survey implementation: Ø costs optimization Ø data quality Ø relations between data quality and costs • Cost effectiveness of web surveys: Ø costs for increasing response rates Ø evaluations of costs in relation to errors

Related issues 2. Survey mode • Web surveys vs. other modes: Ø lower response rates (Lozar Manfreda et. al. , 2007) Ø comparable or higher level of validity and reliability (e. g. Fricker et al. , 2005; Chang & Krosnick, 2002)

Related issues Mixing survey modes Solicitation Interview Contact with the respondent PAPI, CATI, CASI, … personal, telephone, e-mail, … Centralized data management

Related issues Selecting the right mixture Willingness to participate Cost models and impact of environment Response rates and behaviour Structure of the data (MAR) Modelling mixed modes Data quality requirements Optimal combination

Related issues Programing of the questionaire User-friendly interface Paper questionnaire Standardized questionnaire description in QML Internet survey CAPI CATI CASI

Related issues 3. Mixing research methods • Quantitative and qualitative research methods • Flexible combinations of approaches (Morgan, 1998): Ø qualitative preliminary Ø quantitative preliminary Ø qualitative follow-up Ø quantitative follow-up • New possibilities with the Internet

Related issues Flexible on-line mixed methods (FOMM) Analysis Qualitative method Conclusion Beginning Quantitative method

Related issues 4. Ethics, guides and standards • New ethical dilemmas in survey research: Ø solicitation process, spam and privacy Ø data protection Ø children and minors Ø combining data… • Development of new standards (e. g. ESOMAR, Ao. IR, MRA…)

Current and future applications Applications • Different applications according to: Ø type (internet access panels, on-line voting, intercept…) Ø organizations (official, academic, commercial) Ø topics (marketing research, polling opinions, health…) Ø complexity (simple one-shot surveys vs. complex integrated data collection)

Current and future applications Related areas Ø psychological research Ø online experiments Ø paradata and online measurement Ø specific research fields (usability research, HCI. . . ) Ø emerging areas (e-learning, e-government…)

Current and future applications Trends • Expansion of CASIC modes, options and mixtures • Integration of devices (TV, mobile, . . ) • Speech recognition and touch-screens • Multimedia and interactivity • Integration with other CADAC (GPS, paradata, observations, . . . ) • Specialisation and professionalisation