Advancing Paradata and the UK Poverty and Social

Advancing Paradata and the UK Poverty and Social Exclusion survey David Gordon & Eldin Fahmy Townsend Centre for International Poverty Research University of Bristol

Poverty in the UK: Advancing Paradata Analysis and Open Access University of Southampton Ros Edwards Web Site http: //www. poverty. ac. uk Institute of Education Ann Phoenix Heather Elliott Michelle Cage University of Bristol David Gordon Eldin Fahmy Karen Bell Viliami Fifita Web Site http: //www. bristol. ac. uk/poverty

The research aims 1. To improve the measurement of poverty, deprivation, social exclusion and standard of living. 2. To measure the change in the nature and extent of poverty and social exclusion over the past ten years. 3. To produce policy-relevant results about the causes and outcomes of poverty and social exclusion.

Background Every decade since the late 1960 s, UK social scientists have attempted to carry out an independent poverty survey to test out new ideas and incorporate current state of the art methods into UK poverty research. • 1968 -69 Poverty in the UK survey (Peter Townsend and colleagues), • 1983 Poor Britain survey (Joanna Mack, Stewart Lansley) • 1990 Breadline Britain survey (Joanna Mack, Stewart Lansley) • 1999 Poverty and Social Exclusion Survey (Jonathan Bradshaw and colleagues) and its 2002 counterpart in Northern Ireland (Paddy Hillyard and colleagues) • 2012 Poverty and Social Exclusion in the UK

Survey Data Omnibus Survey: Necessities of Life – A systematic random achieved sample of adults (16+) of over 1, 900 interviews in Britain and Northern Ireland. Main Survey: Poverty & Social Exclusion – A follow-up survey to the 2010/11 Family Resources Survey (circa 47, 000 households). The main survey in Britain has an achieved sample of over 4, 000 households and 9, 500 individuals – with approximately 1, 000 households in the ‘ethnic’ strata and 1, 000 households in Scotland. In Northern Ireland, the achieved sample was just under be 1, 000 households and over 2, 000 individuals. The survey was divided into a household questionnaire which was answered by the Household Reference Person (HRP) and an individual questionnaire which was answered by all adult household members (aged 18 and over).

Qualitative Data British Impoverishment Study, The qualitative study, Life on a Low Income in Austere times sought to provide insights into the experiences of poverty during the recent recession and ongoing programme of austerity. The project collected 62 video/audio testimonies during 2012 -2013, in Birmingham, Glasgow and Gloucestershire. Northern Ireland Family Solidarity Study, a qualitative survey of 50 parents to explore the role of family in coping with poverty. The study had a material focus – examining the extent to which resources are transferred among family members (both nuclear and extended) - and it also explores family cultures and relationships as factors affecting poverty and social exclusion. This helps to understand the extent and limits of family solidarity

The Bristol Social Exclusion Matrix (B-SEM) Material & economic resources Health & wellbeing Living environment Crime, harm & criminalisation Social exclusion Economic participation Cultural participation Access to public & private services Social resources Education & skills Social participation Political and civic participation

2012 PSE-UK Question development and testing Focus groups Survey pre-test • • Exploring public perceptions of deprivation, living standards and social exclusion in the UK today to inform the Nat. Cen Omnibus and PSE Main Stage surveys Expert review • Cognitive Interviews • Qualitative pre-testing of selected indicators for potential inclusion in the PSE Main Stage survey Conventional survey pilot with interviewer de-briefing and analysis of non-response Input from team members and UK and International Advisory Groups Survey paradata • Behaviour coding; analysis of question timings and interviewer characteristics

Paradata in the Hong Kong Standard of Living Survey 《香港的生活水平研究》中的環境數據 Paradata is a recent and rapidly expanding area of interest amongst social survey researchers. Given the utility of paradata as a means of assessing survey and data quality, it would be very helpful if you can spend few minutes to fill in the following information recorded for each interview and included on the subsequent survey data file. 在社會調查研究學科內，環境數據（paradata）是一個新興和迅速發展的研究議題。環境數據是 有效評估調查和數據質素的方法，我們希望 閣下能在完成每份問卷調查後花數分鐘時間填寫以下的資 料，并並附於該問卷調查的檔案中。 Interviewer ID code訪問員的身份編碼: ___________ Age of the interviewer 訪問員的年齡: ___________ Gender of the interviewer. 訪問員的性別 Male 男 Female 女 Date of the interview訪問日期: ___________ Time of the interview訪問時間: ___________ Duration of the interview訪問時間的長度: ___________ minutes 分鐘 If the interview was terminated before completion訪問是否中止，未能完成 Yes是 No否

Was the interviewee suspicious about the study? 1. Not at all 2. To some extent, yes 3. Absolutely How do you rate the interviewee's understanding of the questions? 1. 2. 3. 4. 69% 28% 4% 24% 51% 24% 2% Excellent Good Fair Poor Overall, how great was his/her interest in the interview? 1. Very high 2. Above average 3. Average 4. Below average 5. Very low 3% 27% 53% 14% 4% How do you rate the reliability of the information provided by the interviewee about their household’s income? 60% 1. Accurate 36% 2. Fair 4% 3. Inaccurate The PSEHK survey Interviewers thought that only 60% of respondents provided ‘accurate’ information about their household incomes. This could be problematic in a poverty survey if only income information is used to measure ‘poverty’

Why behaviour coding? • No consensus on what is involved in adequate testing survey questionnaires • Pretesting often limited to interviewers perceptions of the response process • Non-response analysis uninformative about potential for response problems not resulting in item non-response • Application of systematic coding framework to survey interview transcripts to identify problems questionnaire administration • Assumes analysis of interview behaviour provides a reliable means of identifying question problems without the need to speak directly to respondents • Assumes that deviations from the paradigmatic, standardised survey interaction are a source of measurement error to be minimised • No consensus on what is coded and how: implications for evaluation of survey questionnaires poorly understood

About the PSE-UK Main stage survey design • Household survey conducted separately in Britain and Northern Ireland by Nat. Cen (Britain) and NISRA. Funded by the ESRC • Sample of respondents in 2010/11 FRS giving permission to be re-contacted. PSE fieldwork conducted Feb. Sept. 2012 using CAPI with all adults within selected households • Target interview duration of 50 mins for single person hhld (+30 mins. for each additional adult. • 23 interviews audio-recorded and transcribed verbatim. Sample selected by the survey organisation from experienced interviewers. Subsequent analysis of 111 question items

Behaviour coding framework • • Observations are Resp*Qitems. In theory 2, 533 observations (23*111) • 1, 689 valid observations in total. Selected respondents provided codeable responses to between 62 and 84 of the 111 items (mean=73. 4; sd=6. 3)

Interview problem codes: frequency counts (%)

Respondent problem codes: frequency counts (%)

Respondent problems by interviewer behaviour Row percentages

Mean number of question problems by interviewer and respondent characteristics

Interviewer and respondent problems by question item: standardised mean scores

Concluding thoughts • How useful (if at all) are different questionnaire pretesting techniques? • To what extent do they give comparable results? What are the main differences in the results they give? NEXT STEPS… • Comparison of behaviour coding with interaction analysis of textual data • Comparison of PSE-UK behaviour coding with: – Cognitive interview question testing – Survey response and question timing paradata – ‘Traditional’ survey pilot results and expert review • How should we explain and respond to inconsistent results between methods?