The following lecture has been approved for University

The following lecture has been approved for University Undergraduate Students This lecture may contain information, ideas, concepts and discursive anecdotes that may be thought provoking and challenging It is not intended for the content or delivery to cause offence Any issues raised in the lecture may require the viewer to engage in further thought, insight, reflection or critical evaluation

Questionnaires & Surveys Delivery Design Development Improving Response rates Prof. Craig Jackson Head of Psychology Division Education Law & Social Sciences BCU

Are Postal Questionnaires Dead Yet? IT predicted death of postal surveys Use of IT at home and work increases survey methods Comparison of surveys using POSTAL WWW or EMAIL or Subjects – UK university staff 200 email questionnaires 200 emails with www url 100 postal questionnaires Asked 3 questions: teeth cleaning fruit walking Jones, R. and Pitt, N. 199

Are Postal Questionnaires Dead? Results Days after sending email n=200 www n=200 post n=100 numbers responding 3 25 5 59 9 61 10 63 17 68 17 day response rate 34% 31 34 35 35 37 19% 0 16 53 60 72 cost per reply Actual cost 35 p With 100% response 19 p 41 p 7. 5 p 92 p 72%

What this means for surveys Golden age of communication? Postal methods still much better rates Novelty value of email is dead - therefore lower response Junk mail perceptions Email Filters are improving Postal letters demonstrate & emphasize their importance www surveys & email allow immediate data processing (software) Email & www have potential for low cost regular user surveys

Introduction Questionnaire is a fundamental component of most research projects Most MSc / Ph. D projects use questionnaire methods Largest reason for criticising projects - weak / dubious questionnaires Can be very efficient A Questionnaire does: Take planning to get right Reward time spent on it Capture information A Questionnaire is not: Constructed in 10 minutes An easy option A collection of simple questions

Basics of Measurement tools must be appropriate Psychometrics -- personality, attitudes, stress, symptoms, Physical measurements -- working, environment, symptoms Exposure assessments -- hazards, risks, ppm 3, duration When measuring. . . Take multiple measurements (and take the mean) Under same conditions, but if not…. Statistical remedies to adjust e. g. age, time of day etc. Reliable & Validated tool Defined and Regular variables

Dilemma Achieving a high response rate to a questionnaire is vital But does not promise a normal distribution of responses? Postal questionnaires rarely get a response rate > 40%. Unless respondents have a vested interest in the outcome. Bias? Most efficient (best) response rates usually happen when respondents have to do very little to take part in the study. Multiple phase projects see a depletion in numbers at every stage. Quick “in and out” one-stop approach is best http: //onlinestatbook. com/stat_sim/sampling_dist/index. html

Know your enemy

Diminishing returns of multi-stage recruitment Researcher Potential Sample 1000 people 540 consents 540 questionnaires Under-powered study n = 210 Response rate of 21% 210 questionnaires

Structure Identifying Items Title Preamble Branch Instructions Research Items does not need to be honest some deception is necessary shorter is better

Identifying items Preliminary questions Collecting info necessary for screening: recording keeping tracking tracing data manipulation Ask only for relevant info - unethical Fewer items minimizes chance of alienating respondent The need for rich info to improve the study Participant’s need for privacy & anonymity

Encryption devices Steganography Secret communication of a message by hiding it’s existence Steganos, meaning covered. Gk Graphein, meaning to write, Gk If message is discovered it is easily read because of no encryption Cryptography Secret communication of a message by hiding it’s meaning Kryptos, meaning hidden. Gk Message established using a known protocol, to be decrypted by the receiver Steganography & Cryptography can be combined together if

Encryption devices Steganography “This example of steganogrpahy may not work very well when projected onto a large screen, but it works very well on paper, such as questionnaires. Hopefully many of you will notice the method of steganography used in this piece of text. Gosh how clever I am……” Cryptography drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui The above text (containing an encrypted name and address) looks suspicious and may be obliterated by the respondent drbjh kbdltpo, jotujuvuf pg pddvqbujpobm

Preamble An important introduction Frame of reference Without a FOR a respondent may base their answer on a wrong context “We would like to know if you have had any medical complaints and how your health has been in general, over the past few weeks. Please answer ALL the questions on the following pages by ticking the answer which best applies to you. Remember that we want to know about present and recent complaints, not those that you had in the past. It is important that you try to answer all questions”

Title Not to be underestimated – try to be user-friendly Sets the tone for the respondent The General Health Questionnaire vs The GHQ 28 “Simple language for simple people” Beware of abbreviated titles - may alienate respondents e. g. Agricultural Satisfaction Scale LEI BDI MMPI HAD etc

Research items Set(s) of items to collect the “real” research data Take many forms: Item Type Data type Fixed option items (ordinal or nominal) Rating scale items (likert) Yes / No items (binary) True / False items (binary) Likert scale items (likert) Diagrams binary, ordinal, or nominal

Branch Instructions Guide the respondent through the items Maximises the efficiency of respondent’s efforts Avoids redundant items Focuses on worthwhile items Instructions can sometimes confuse the respondent so think carefully e. g “If you answered ‘NO’ to question 9, ignore 9 b, 9 c, and 9 d, go to question 10. ”

Ah, the Truth

Are your respondents telling the truth? Use a sprinkling of LIE DETECTOR items to assess reliability e. g Respondent “Outrage” increases “Did you ever lie to your parents as a child? ” with item intimacy “Have you ever visited a pornographic web-site? ” “Have you ever taken anything without asking permission? ” Decide what to do with any respondent who “fails” the lie detector Keep them in? Exclude immediately? Retain but “flag” them?

Funnel Items Set funnel questions to try to narrow down a respondent to specific details e. g 6 a “Did you write essays in college? ” Yes / No 6 b “What type of essays were they mostly? ” Narrative / Descriptive / Persuasive 6 c “How often did you write them? ” ___ per term

Response Set Bias Enables collection of detailed data easily Build a detailed data set from a simple binary item Randomise the layout / order of responses “true / false” e. g True True False False Respondents ticking the same items Swap the “True & False” responses sequence

Likert scales A visual linear scale for rating purposes happy X sad Swap around the words to avoid response-set bias Avoid using numbers on the scale -- can be (mis)leading

Likert scales XX X X happy XX X X XXX X X sad

Likert scales XX X happy XX X X X sad

Design Pointers Avoid colloquialisms or abbreviations Beware of local expressions Avoid words with double meanings e. g. ”Fair. . Dip. . Lie. . Well” Set a definition of specific terms e. g. “OK. . . Average” Avoid long questions Specify exact time, place and context ever. . . ” e. g “At school, did you Phrase items to make denial impossible e. g “When did you first. . . ” Avoid numbers on your responses / scales

Design Pointers Clearly phrase items Make items unambiguous Avoid leading items think. . . ” e. g. “Many people Ask only what a respondent is qualified to answer Avoid socially loaded items or “junkies” e. g. “beggars” Do not use socially / religiously biased items Avoid phrases, clichés or sooth-sayings Ask for ages as a true number / D. O. B - - NOT in age groups Font Pictures

Most common errors “In your organization, do women have the same responsibilities as men, and should women have more? ” “Preventing accidents in the workplace is vital, and more money should be spent on prevention. ” “Training in risk assessment is something I would like to do, and I would like to see my colleagues do it to. ” Only ask one question per item

Most common errors What is your age ? (please tick) 16 - 26 26 - 36 36 - 46 46 - 56 What is your marital status ? (please tick) Married Single How many blood splashes have you had ? (please tick) 1 -5 6 - 10 11 - 15 15 - 20 20 plus Are life preservers and flares essential on-deck equipment ? agree disagre

Factorial approach The GHQ 28 A self-completion questionnaire assessing mental health How? 28 items 7 about Anxiety 3 7 about Severe depression 7 about Dysfunction 7 about Somatic symptoms Anxiety score Depression score Dysfunction score Somatic score Global score 4 5 2 14 By summing the 4 factors there is a Global Mental Health score Statistics are performed on the factor scores and the global score

Designing factors Piloting items Set down as many items as possible concerning the topic Limit the questionnaire to 30 items Give the questionnaire to at least 20 people Score responses and place into SPSS spreadsheet Factor Analysis use statistics to “group” items together into factors (will do this as a group exercise in the SPSS lecture)

Creating factors Factor Analysis Results in manageable number of factors instead of 30 items Each factor comprised of a number of items that had similar answers from your pilot sample The similarity of responses to particular items is what SPSS uses to group items together into factors Identifying Items As a matter of routine: Get as much information as possible may be vital in later (unforeseen analyses) Sex DOB or Age (not age groups) Marital status / Domestic arrangements

Increasing Response Rates Incentives Appearance Delivery Origin Contact Content Communication

Increasing Response Rates - Incentives Money Vouchers Prize draw Ethical aspects Bias O. R Monetary incentive vs. None 2. 02 Incentive with Q. vs. Incentive on return 1. 71 Non-monetary incentive vs. No incentive 1. 19 Edwards et al. 2002

Increasing Response Rates - Appearance O. R Shorter format vs. Longer format Brown envelope vs. White 1. 86 1. 52 Coloured ink vs. Black 1. 39 Folder / Booklet vs. Stapled pages 1. 17 Personalised vs. Not personalised 1. 16 ID feature on return vs. No ID 1. 08 Coloured Q vs. White Q 1. 06 Edwards et al. 2002

Increasing Response Rates – Delivery methods O. R Recorded delivery vs. Standard 2. 21 Stamped return envelope vs. Business reply / franked 1. 26 Q sent to work vs. Q sent to home 1. 16 1 st class outward mail vs. Other class 1. 12 Pre-paid return envelope vs. Not pre-paid 1. 09 Stamped outward envelope vs. Franked 0. 95 Commemorative stamp vs. Ordinary stamp 0. 92 Edwards et al. 2002

Increasing Response Rates – Origin & Contact Origin O. R University vs. Other organisation 1. 31 Sent by senior persons vs. Juniors 1. 13 Ethnically ambiguous name vs. Non-white name 1. 11 Contact O. R Pre-contact vs. No contact 1. 54 Follow up vs. No follow up 1. 44 Postal follow up with Q vs. Without Q 1. 41 Mentioning follow up vs. None 1. 04 Pre-contact by telephone vs. Postal pre-contact 0. 90 Edwards et al. 2002

Increasing Response Rates - Content O. R More interesting vs. Less interesting User-friendly vs. Standard 2. 44 1. 46 Factual items only vs. Factual & attitude items 1. 34 Relevant items first vs. Other items 1. 23 Demographic items first vs. Other items 1. 04 “Don’t know” boxes vs. no “Don’t know” boxes 1. 03 Sensitive items vs. No sensitive items 0. 92 General items first vs. Last Edwards et al. 2002 0. 80

Increasing Response Rates - Communication O. R Explain drop-out required vs. Not 1. 32 Stresses benefit to respondent vs. Others 1. 06 Stresses benefit to sponsors vs. Others 1. 01 Stresses benefit to society vs. Others 1. 00 Response deadline given vs. No deadline 1. 00 Instructions given vs. No instructions 0. 89 Choice to opt out given vs. No opt out 0. 76 Edwards et al. 2002

Questionnaire Summary 1. Postal questionnaires widely used in data collection 2. Mark pre-pay or addressed envelopes 3. Use steganography over cryptography 4. Perform analyses on factors not individual items 5. Identifying items may be useful in later analyses 6. Think about scoring items before rushing ahead with the questionnaire 7. Store questionnaires securely until passing any viva voce

Questionnaire Summary cont. 9. Perfect for epidemiological studies and health research 10. Non-response to postal questionnaires reduces sample & introduces bias 11. Identification of effective ways to increase postal response rates 12. Use existing metrics - pilot items if making new questionnaire 13. Use factor analysis 14. Keep it all as brief as possible