Design Method of Data Collection Surveys and Polls

Design Method of Data Collection Surveys and Polls Observational Experimentation Studies C 1, L 2, S 1

Sampling/Chance/ Random Errors Nonsampling Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects C 1, L 2, S 2

Sources of Nonsampling Errors Selection bias Population sampled is not exactly the population of interest. e. g. KARE 11 poll, telephone interviews sample population C 1, L 2, S 3

Sources of Nonsampling Errors Non-response bias People who have been targeted to be surveyed do not respond. Non-respondents tend to behave differently to respondents with respect to the question being asked. C 1, L 2, S 4

1936 U. S. Election • Country struggling to recover from the Great Depression • 9 million unemployed • 1929 -1933 real income dropped by 1/3 C 1, L 2, S 5

1936 U. S. Election • Candidates: – Franklin D Roosevelt (Democrat) Deficit financing - “Balance the budget of the people before balancing the budget of the Nation” – Albert Landon (Republican) “The spenders must go!” This man has never been President. C 1, L 2, S 6

1936 U. S. Election • Roosevelt’s percentage – Digest prediction of the election result 43% – Gallup’s prediction of the Digest prediction 44% – Gallup’s prediction of the election result 56% – Actual election result 62% • Digest sent out 10 million questionnaires to people on club membership lists, telephone directories etc. – received 2. 4 million responses • Gallup Poll used another sample of 50, 000 • Gallup used a random sample of 3, 000 from the Digest lists to predict Digest outcome C 1, L 2, S 7

Sources of Nonsampling Errors Self-selection bias People decide themselves whether to be surveyed or not. Much behavioural research can only use volunteers. C 1, L 2, S 8

Sources of Nonsampling Errors C 1, L 2, S 9

Sources of Nonsampling Errors This poll is not scientific and reflects the opinions of only those Internet users who have chosen to participate. The results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole. The Quick. Vote sponsor is not responsible for poll content, functionality or the opinions expressed therein. C 1, L 2, S 10

Sources of Nonsampling Errors Question effects Subtle variations in wording can have an effect on responses. e. g. “Should euthanasia be legal? ” vs. “Should voluntary euthanasia be legal? ” C 1, L 2, S 12

18 August 1980 New York Times/CBS News Poll “Do you think there should be an amendment to the constitution prohibiting abortions? ” Yes 29% No 62% Later the same people were asked: “Do you think there should be an amendment to the constitution protecting the life of the unborn child? ” Yes 50% No 39% C 1, L 2, S 13

Question Effects in the NZ Census 1986: “What is your ethnic origin? (Tick the box or boxes which apply to you. ) 1991: “Which ethnic group do you belong to? ” (Tick the box or boxes which apply to you. ) 1996: “Tick as many circles as you need to show which ethnic group(s) you belong to. ” Ethnicity Single Ethnicity European Maori Two Ethnicities European & Maori Two European gps 1986 94. 6 1991 94. 3 1996 81. 0 81. 2 9. 1 4. 0 2. 9 0. 0 78. 1 9. 6 4. 5 2. 7 0. 6 65. 8 7. 6 11. 2 4. 7 4. 5 C 1, L 2, S 14

Sources of Nonsampling Errors Interviewer effects Different interviewers asking the same question can obtain different results. e. g. the sex, race, religion of the interviewer C 1, L 2, S 15

Interviewer Effects in Racial Questions In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked: “Do you personally feel that you trust most white people, some white people or none at all? ” • White interviewer: 35% answered “most” • Black interviewer: 7% answered “most” C 1, L 2, S 16

Sources of Nonsampling Errors Behavioural considerations People tend to answer questions in a way they consider to be socially desirable. e. g. pregnant women being asked about their drinking habits C 1, L 2, S 17

Behavioural Considerations in Election • Official vote counts show that 86. 5 million people voted in the 1980 U. S. presidential elections. • A census bureau survey of 64, 000 households some weeks later estimated 93. 1 million people voted. C 1, L 2, S 18

Sources of Nonsampling Errors Transferring findings Taking the data from one population and transferring the results to another. e. g. Minneapolis opinions may not be a good indication of Winonan opinions. Minneapolis sample Winona C 1, L 2, S 19

Sources of Nonsampling Errors Survey-format effects e. g. question order, survey layout, interviewed by phone or inperson or mail. C 1, L 2, S 20

Sampling/Chance/ Random Errors Nonsampling Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects C 1, L 2, S 21

1999 New Zealand Newspaper Graphic C 1, L 2, S 22

Nonsampling Errors What nonsampling errors could be present if the results of the survey were used as representative of the views held by New Zealanders in general? Discuss in groups for 1 minute. C 1, L 2, S 23

Nonsampling Errors • Selection bias – only those who read this paper – only those that can afford to pay for the call • Question effects – how do you answer the question? ? • Self-selection bias C 1, L 2, S 24

Survey Errors Sampling/Chance/ Random Errors Nonsa mpling Errors C 1, L 2, S 25

Sampling / Chance / Random Errors • errors caused by the act of taking a sample • have the potential to be bigger in smaller samples than in larger ones • possible to determine how large they can be • unavoidable (price of sampling) C 1, L 2, S 26

Nonsampling Errors • can be much larger than sampling errors • are always present • can be virtually impossible to correct for after the completion of survey • virtually impossible to determine how badly they will affect the result • must try to minimize in design of survey (use a pilot survey etc. ) C 1, L 2, S 27

Surveys / Polls A pilot survey is a small survey that is carried out before the main survey and is often used to identify any problems with the survey design (such as potential sources of non-sampling errors). C 1, L 2, S 28

Surveys / Polls A report on a sample survey/poll should include: – target population (population of interest) – sample selection method – the sample size and the margin of error – the date of the survey – the exact question(s) being asked C 1, L 2, S 29