Immigrant respondents and quality in population surveys sampling

Immigrant respondents and quality in population surveys: sampling, non-response & questionnaire design Q 2014, Vienna, 2. 6. 2014 Liisa Larja, Ada Kotilainen

Content of the presentation 1. 2. 3. 4. Survey design Field work Questionnaire design Conclusions

Problem 1: immigrants are underrepresented in the general population surveys 7% 6% 5% Share of foreign born of the population (sample frame) 4% 3% LFS: Share of foreign born from the achieved sample 2% 1% 0% 2007 2008 2009 2010 2011 2012 2013

Problem 2: Low response rate among immigrants decreases the quality of general population surveys 73, 0 % 71, 0 % 69, 0 % 67, 0 % Total response rate: immmigrants 50 % and natives 72 % 65, 0 % 63, 0 % 61, 0 % 59, 0 % Level 2013: Share of 6, 6 % of immigrants the sample x 2 x 3 x 4 are immigrants Total response rate: immmigrants 72 % and natives 72 %

Problem 3: Disaggregating results according to immigrant status is not possible when there are too few immigrants in the general population survey samples No results: problems go unnoticed. Separate studies: comparison? n EU immigrant integration indicators derive from Labour Force Survey (LFS), Survey on Income and Living Conditions (SILC) and PISA n LFS ad hoc module 2014 on Labour market situation of immigrants and their descendants in the Finnish LFS the number or immigrants would be very low (~900). n Oversampling – expensive n Pooling of data from several years – timeliness and quality problems, not possible for LFS ad hoc module 2014 n

Solution: Survey design where the data for 3 surveys is collected within 1 interview

The ”UTH”-study (Survey on work and well-being among persons of foreign origin) N = 5 400 individuals of ”foreign origin” (= both parents foreign-born) all languages, all areas, citizens, foreigners, 2 nd generation…. n Sampling from population register. At least 6 % overcoverage: moved out from Finland n Computer assisted personal interviews (CAPI). Normally done in respondents home, but 50 % of the respondents wished for some other place. n Standardized interview technique n

2. Field work For the first 7 reference weeks the response rates have been over 70 % , which is slightly better than in corresponding general population surveys. n Translating questionnaires and programming them into Blaise also “monolingual” interviewers were able to use language versions n Translating cover letters and info sheets on data secrecy n Recruiting 9 multilingual interviewers (through NGO’s) in addition to 120 permanent interviewers n

…field work Training, motivating & encouraging interview staff (sensitive questions, interpretation, motivating different types of respondents) n Investing in searching for missing contact information (different registers, home visits & notes, neighbours, googling, social media, e-mail, SMS). Average time spent per sampling unit 3, 5 hours (interview 0, 7 h). n Non-response due to same reasons as for general populations: non-contacts, no time, no interest. n

3. Questionnaire testing: an iterative process n n n 21 cognitive interviews (incl. one proxy). Focus on new questions. How do the respondents interpret the questions, how do they choose their response, do they leave something unsaid? 6 pilot interviews (incl. one ad hoc translation). Testing the entire questionnaire. Length and flow of the interview, spontaneous reactions to the questions. 21 pilot interviews. Testing of languages versions, testing of distant translation by phone and video (+ length and spontaneous reactions)

Questionnaire design: Comprehension problems Linguistic problems (difficult words and sentence structures, Likert scales) incorrect interpretation of the questions reformulation of the questions l Many misinterpretations emerged only in the cognitive interviews. n Standardized scales “easy Finnish” versions + help-text, to decrease need to deviate from standardized interview technique n Unfamiliarity with the interaction pattern of structured interview The length of the questionnaire was cut by 40 % Investing in language versions. n

Questionnaire design: translations Finnish, Swedish and English + 9 additional languages. 68 % had language version in their mother tongue n The number of language versions was prioritized over quality of translation: l Professional translation, but no back-translation, committee translation etc. l Interviewers proof-read the questionnaires and only obvious mistakes were corrected. n With only a few (rigorously tested) language versions there would be even more misunderstanding due to broken Finnish/English as compared to linguistic differences due to differences in translations. n

Questionnaire testing: trust Questions on reasons for migration, language skills, cultural identity, labour market obstacles, health problems, experiences of violence, pregnancies etc. were found sensitive, especially with proxy respondent or when a friend was assisting with translation routing was changed to skip some of these questions for proxy respondents and when there were other persons assisting in the interview response cards were added to allow responding without speaking n The respondents expressed concerns on whether the information they give may have an effect on decisions on their social benefits etc. info sheet on data secrecy n The respondents admitted, that the imagined opinion of the interviewer influenced option selection interviewer training on sensitivity n

Conclusions Oversampling immigrants can be done cost-effectively by collecting data for multiple surveys during the same interview n High response rates can be achieved by investing in quality in field work (language version, working time in contacting) n Questionnaire testing is essential and it is best done iteratively n Finding solutions to the problems of comprehension, and trust is essential as they affect response behaviour and produce bias in the results. n