EMR 6500 Survey Research Dr Chris L S

EMR 6500: Survey Research Dr. Chris L. S. Coryn Lyssa N. Wilson Spring 2015

Agenda • The tailored design method • Coverage and sampling • Case Study #1

The Tailored Design Method

The Tailored Design Method • Uses multiple motivational features in compatible and mutually supportive ways to encourage high quantity and quality of responses

The Tailored Design Method • Premised on social exchange perspective on human behavior • Assumes that the likelihood of responding is greater when the expected rewards outweigh the anticipated costs

The Tailored Design Method • Gives attention to all aspects of contacting and communicating with respondents • Encourages response by considering survey sponsorship, the nature of the population and variations within it, and content of questions

The Tailored Design Method • Emphasizes reducing errors of coverage, sampling, nonresponse, and measurement

Coverage Error • Occurs when all members of a population do not have a known, non -zero probability of selection • Occurs when those who are excluded are different from those who are included

Sampling Error • Results from surveying only some rather than all members of a population • Represented by B, the bound on the error of estimation

Nonresponse Error • Occurs when people selected do not respond are different than those who do • Nonresponse can occur at the level of items within a survey or at the level of the survey – MAR – MCAR

Measurement Error • Occurs when responses are inaccurate or imprecise • Primarily related to poor layout and poor design and wording of questions

Competing Perspectives • Economic exchange view of survey response • Psychological models of survey response • Leverage-saliency theory of survey response • Social exchange theory of survey response

Economic Exchange • Use monetary rewards as the primary motivation for seeking responses • Widely adopted, especially in panel surveys

Psychological Models • Extrinsic and intrinsic considerations motivate respondents • Guided by social psychological concepts such as scarcity of opportunity, consistency with previous behavior, desire to reciprocate, enjoyment of task, and social proof

Leverage-Saliency Theory • Respondents are differentially motivated by different aspects of the survey (leverage) and by how much emphasis is placed on each aspect by the surveyor (salience) • Overemphasis on a single appeal that is attractive to some is not to others

Social Exchange Theory • Premised on actions being motivated by the return that actions are expected to bring from others • Simply, rewards are greater than costs

Social Exchange and Surveys • Addresses three central questions about design and implementation 1. How can the perceived rewards for responding be increased? 2. How can the perceived costs of responding be reduced? 3. How can trust be established so that people believe the rewards will outweigh the costs of responding?

Increasing Benefits • • • Provide information about the survey Ask for help or advice Show positive regard Say thank you Support group values Give tangible rewards Make the questionnaire interesting Provide social validation Inform people that opportunities to respond are limited

Decreasing Costs • Make it convenient to respond • Avoid subordinating language • Make the questionnaire short and easy to complete • Minimize requests for personal or sensitive information • Emphasize similarity to other requests or tasks to which a person has already responded

Establishing Trust • Obtain sponsorship by legitimate authority • Provide a token of appreciation in advance • Make the task appear important • Ensure confidentiality and security of information

Features that can be Tailored • Survey mode – Singular or multiple • Sample design – Type of sample – Number of units sampled • Incentives – Type of incentive – Amount or cost of incentive – Before or after

Features that can be Tailored • Contacts – Number of contacts – Timing of initial and subsequent contacts – Mode of each contact – Whether contacts will be personalized – Sponsorship information – Visual design of each contact – Text or words in each contact

Features that can be Tailored • Additional materials – Whether to provide them at all – Type of materials (e. g. , research report) – Visual design of materials – Text or wording of materials

Features that can be Tailored • Questionnaire – Topics included – Length (duration, number of pages/screens, number of questions) – First page or screen – Visual design – Organization and order of questions – Navigation through questionnaire

Features that can be Tailored • Individual questions – Topic (sensitive, of interest to the respondent) – Type (open-ended versus closed-ended) – Organization of information – Text or wording – Visual design

Coverage and Sampling

Central Terminology • An element is an object on which a measurement is taken • A population is a collection of elements to which an inference is made from a sample • A sample is a collection of sampling units drawn from a frame or frames • Sampling units are nonoverlapping collections of elements from the population that cover the entire population • A frame is a list of sampling units

Central Terminology • A completed sample is the units that respond • Sampling error is the result of collecting data from only a subset, rather than all, units from a frame – Again, represented by B, the bound on the error of estimation

Coverage • The degree to which the units in a sampling frame correspond to the population of interest • Coverage is likely one of the most serious problems in most surveys

Coverage and Frame Problems

Telephone Coverage • Predominant sampling frame in the 1980 s and 1990 s – Random digit dialing (RDD) • Since the introduction of the cellular telephone valid coverage is no longer possible – Approximately 18% of all households no longer have a landline – Differences between cellular phone users and traditional landline users

Internet Coverage • Significant coverage gaps in the general population – Approximately 70% of the population has high-speed internet access in their homes – An increase from 47% in 2007 • Widely used for specific, targeted populations (e. g. , students, professionals) • No equivalent to the RDD algorithm

Mobile Technology • 90% of American adults have a cell phone • 58% of American adults have a smart phone • 42% of American adults own a tablet computer

Mail Coverage • As with telephone, widely used until the 1990 s • Increasingly unlisted telephone numbers (and addresses) • Changes in social norms (e. g. , doublelisting of spouses with different last names) • Address-based sampling using U. S. Postal Service DSF (all delivery point addresses) – Can be geo-coded

Reducing Coverage Error • Most surveyors are interested in specialized subpopulations rather than the general population • In certain instances, valid sampling frames can be established

Reducing Coverage Error • Central questions: – Does the list contain everyone in the survey population? – Does the list include people who are not in the study population? – How is the list maintained and updated? – Are the sample units included on the list more than once? – Does the list contain other information that can be used to improve the survey?

Respondent Selection • Should be carefully coupled to the focal question of the study – Most recent birthday method (if interest is in adult population) – Greatest responsibility (if interested in household behaviors)

Coverage Outcomes • Careful coverage analysis – Multiplicity • Duplicate units in the sampling frame – Overcoverage • Units included in the sampling frame that are not in the target population • Units that do not meet inclusion criteria – Undercoverage • Units that are not in the sampling frame but that are part of the target population • Units that meet inclusion criteria

Probability Sampling • Only method that allows the statistical properties of estimators to be assessed probabilistically • Always the preferred method for sampling, even in small finite populations

Sample Size • It is the size of the sample, not the proportion of a population sampled, that determines precision

Basic Rationales • Relatively few responses can provide precise estimates • In large populations there is virtually no difference in the number of sampled units needed for a given level of precision • In small populations greater proportions need to be sampled for a given level of precision • In large samples additional increases yield small reductions in error • Sample sizes need to be larger if interest is in subpopulations

Case Study #1

Case Study Activity • In small groups, address the following questions in relation to Case Study #1 relying only on the material that was discussed in today’s lecture and readings 1. Has the surveyor committed any serious error(s)? 2. If so, what type and why? If not, why?