Mining Interviewer Observation Daniel Guzman Statistics and Methods

Problem • Gaining cooperation of survey respondents is getting more and more difficult. (Curtis, Presser, and Singer 2005; Groves 2006; Brick and Williams 2013) © 2018 by the Regents of the University of Michigan

Approaches • • Send advance letters Offer monetary or non-monetary incentives Send reminders or making follow-up calls Tailored designs based on the main factors preventing survey participation PARADATA © 2018 by the Regents of the University of Michigan

Interviewer observations • Different types: – Housing observation – Contact observation – Interview observation – Call notes • Interviewer observations are a rich source of information, but they can be hard to analyze. • Open text, unstructured, prone to errors © 2018 by the Regents of the University of Michigan

Data • Longitudinal panel study of persons age 51+ • New birth cohort every

Response Rate Cohort 2004 2006 2008 2010 2012 2014 Early Baby boomers 75. 3%

Analysis • Call notes from attempts resulted in contact with informant/respondent before the final

Example of Observation iwer knocked and son answered the door. He appears to be over 18. iwer started to introduce myself and he stated he recognized me and laughed. I asked for the R and again she is sleeping. i asked if there was a good time to come back today and he stated no she will get up in a few minutes and go straight to work. Iwer mentioned that the TOA had increased substantially and that I would love to tell her about it but the number I have says the vm box is full. He still would not verify the phone number. i left a SIMY with my umich number on the back. © 2018 by the Regents of the University of Michigan

Sentiment: Panel (1) © 2018 by the Regents of the University of Michigan

Sentiment: Panel (2) © 2018 by the Regents of the University of Michigan

Sentiment: New (3) © 2018 by the Regents of the University of Michigan

Sentiment: New (4) © 2018 by the Regents of the University of Michigan

Model: Predict completion © 2018 by the Regents of the University of Michigan

Model: Sentiments © 2018 by the Regents of the University of Michigan

Topic Analysis • There are two estimates: 1. how much each topic contributes to

Method • Latent Dirichlet Allocation (LDA) – unsupervised method of doing topic modeling. – The tricky part is that words can also belong to more than one topic. • Every document is a mixture of topics • Every topic is a mixture of words © 2018 by the Regents of the University of Michigan

Word Frequency © 2018 by the Regents of the University of Michigan

© 2018 by the Regents of the University of Michigan

Summary • Lack of an appropriate lexicon in the survey context. • Select “stop words” is tricky • Bounded by how and what interviewers decided to included in the observation. • We might need a supervised ML method for topic modeling • Sentiment Analysis might help to predict completion © 2018 by the Regents of the University of Michigan

References • Brick, J. Michael and Douglas Williams. 2013. “Explaining Rising Nonresponse Rates in Cross. Sectional Surveys. ” ANNALS of the American Academy of Political and Social Science 645: 36 -59. • Curtin Richard, Presser Stanley, Singer Elinor 2005. “Changes in telephone survey nonresponse over the past quarter century. ” Public Opinion Quarterly 69 (1): 87– 98. • Groves Robert M. 2006. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly 70 (5): 646– 75. © 2018 by the Regents of the University of Michigan