International Standards for Survey Quality A Total Survey

International Standards for Survey Quality: A Total Survey Error Solution? Paul P. Biemer Lars E. Lyberg www. rti. org RTI International is a trade name of Research Triangle Ins

Aside…Estimator of the MSE Using a Gold Standard is biased is unbiased (gold standard) See equation 2. 20, Chapter 2, Overview of Design Issues: Total Survey Error in Marsden, P. and Wright, J. (2010). Handbook of Survey Research, Emerald Publishers www. rti. org

The Premise of this Paper • There are many standards for surveys; examples include – – ISO 20252 for Market, Opinion and Social Research OMB standards NCES statistical standards Quality guidelines developed by specific organizations (Stat Can; RTI; etc. ) – ESS Code of Practice • Do these standards help in reducing total survey error for surveys? • The answer to this question is not clear. 3 www. rti. org 11/25/2020

Definitions: policies, standards, guidelines and recommended practices. • standards - A practice that is followed almost without exception. Deviations are not recommended and require approval of senior management. Corrective action should be taken when a standard is not being met. ” • policies (no exceptions under any circumstances) • guidelines (can be skipped for “good” reasons), and • recommended practices (promoted but not obligatory) – Colledge and March (1997) www. rti. org

What are survey standards? • A document that – describes methods and procedures for the collecting, processing, storing, and presenting survey data. – define the (minimal) level of quality and effort that is acceptable for all survey processes • May address only some survey processes; e. g. , standards for writing survey reports • Should be auditable; vague standards have no effect on survey quality 5 www. rti. org 11/25/2020

What purposes do survey standards serve? • Communicates a level of quality that organizations establish as minimal for all surveys. • Provide consistency across surveys in different organizations • Facilitate communication of complex concepts, formulas, procedures and methodologies • Provide transparency of the methodologies used to produce a survey data set. • Transfers skills and knowledge of best survey practice 6 www. rti. org 11/25/2020

Some Problems with Survey Standards • Comprehensive standards can overwhelm survey practitioners • Unless mandated and audited, they are largely ignored. • Must be continually updated. • May conflict with client requirements. • Can be costly to develop, maintain and enforce. • No documented evidence exists that they improve data quality. www. rti. org

Idea advocated in this presentation • General survey standards have their strengths and weaknesses, but their effects on data quality have not be documented • Instead we advocate Standards for Survey Quality Investigations (SSQIs) • SSQI’s are like survey standards in that they describe methods and procedures. • SSQIs focus on methodological studies and survey quality investigations. – Example: the AAPOR standard for computing response rates. 8 www. rti. org 11/25/2020

What purposes would SSQIs serve? • To promote data quality investigations. • To encourage sharing of the results of survey methodological studies, including the data themselves. • To communicate the methods and the quality metrics used. • To facilitate methodological meta-analysis and comparative analysis • To increase transparency and consistency • To share skills and knowledge regarding survey quality investigations. • Decrease process variation www. rti. org 11/25/2020 9

Some areas where SSQIs would be beneficial • Computation of frequently used quality indicators – weighted and unweighted response rates (e. g. , AAPOR) – eligibility rates – tracing rates – frame coverage, duplication, and erroneous inclusion rates – design effects (unequal weighting and clustering) – keying, coding and editing error rates – Item nonresponse rates – reliability measures 10 www. rti. org 11/25/2020

Some areas where SSQIs would be beneficial (cont’d) • Reporting standards for these quality indicators • Best methods for collecting data to compute these quality indicators • Best practices for – Nonresponse followup studies, including two phase sampling methods – Investigations of interviewer variance – Evaluations of frame coverage bias – Reinterview studies – Experimentation with incentives 11 www. rti. org 11/25/2020

Resurrecting an old ITSEW idea: Survey Quality Wiki • “Wiki” – Hawaiian for “fast; ” “backronymed” to stand for “what I know is” • Website that allows the easy creation and editing of any number of interlinked web pages via a web browser using a simplified text editor • Typically powered by “wiki software”to create collaborative websites for knowledge management. • Users use their editorial rights to remove material that is considered "off topic. " Such is the case of the collaborative encyclopedia Wikipedia. 12 www. rti. org 11/25/2020

Questions for Discussion • How does the group feel about survey standards in general? – How well do they serve their stated purposes? – Are there good examples of cases where they have improved survey quality? • How does the group feel about SSQIs? Would such standards benefit survey quality research? • What types of SSQIs would be the most beneficial for survey research? • How should SSQIs be promulgated? Is a Survey Quality Wiki a good idea? Is there interest in developing it? 13 www. rti. org 11/25/2020