Quantitative Data Analysis Collecting data with questionnaires Dr

Quantitative Data Analysis: Collecting data with questionnaires Dr. Michelangelo Misuraca Research Assistant Professor in Statistics (Università della Calabria) >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 01 – How to collect data: an introduction There is a wide range of technical solutions for data collection, and the choice of the suitable one depends on several factors extension of the pursuing aims dimension of the target population desired degree of reliability available budget (and time) for data collection OBSERVE/MEASURE a direct way for studying the overt behaviours and the world around us INTERVIEW/DISCOVER a way for exploring motivations, expectations, attitudes, believes, opinions >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 02 – Census or Sample Survey? Sample Population The aim is to know exactly the analysed phenomenon The aim is to "estimate" the analysed phenomenon >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 03 – Precision and accuracy Precision (in Statistics) is directly proportional to the number of analysed entities (reproducibility) Accuracy (in Statistics) is inversely proportional to the number of analysed entities (veracity) low accuracy poor precision high accuracy poor precision low accuracy good precision high accuracy good precision The two concepts can be seen as different dimensions of the reliability >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 04 – Pros and cons Time s labor surveys are les census intensive than Costs eaper surveys are ch Validation results can the survey ? be generalised Richness eeply in d o g n a c s y e v sur omena n e h p g n ti a g ti s inve >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 05 – A classical but relevant case… Candidates: F. D. Roosevelt, A. Landon Literary Digest Poll 10 ml of vote ballots mailed to a sample primarly drawn from automobile registration lists and telephone books => vote ballots returned > 2 ml ! Prediction: Gallup Poll Roosevelt 43% Landon 57% 50. 000 people quota sample interviewed by pollsters across the country Prediction: Roosevelt 56% Actual result… >>> Nottingham Business School – 2013 <<< Landon 44%

Quantitative Data Analysis: Collecting data with questionnaires 06 – The dark side of the poll - In 1936 those who owned cars or had telephones tended to be more educated, have higher incomes (this during the Depression) and be Republican! - The returned ballots came from a given part of the sampled population… what about the nonrespondents? 12 years later… A bigger sample produces more precise estimes but… a random sample is better that a haphazard one! >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 07 – Data Collection methods DIRECT PRIMARY INDIRECT SECONDARY COLLECTION INDIRECT REGISTER ANALYSIS OBSERVATION/ INSPECTION Each of these methods has both advantages and disadvantages SELF-ADMINISTERED ON THE SPOT QUESTIONNAIRES ADMINISTERED TELEPHONE INTERVIEWS FACE TO FACE INTERVIEWS POSTAL QUESTIONNAIRES INDIVIDUAL QUESTIONNAIRES WEB BASED QUESTIONNAIRES FOCUS GROUPS (UNSTRUCTURED) >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 08 – Asking questions with a purpose Questionnaire design is not easy, it requires many different skills and a pinch of experience… It allows the communication among the different actors of the collecting process researchers interviewers respondents I’ve inven ted it ! Researchers must always be prepared to ask themselves ‘Is this question really needed? ’ The shorter the questionnaire the better: questions should be included only if that are believed to be essential >>> Nottingham Business School – 2013 <<< Sir Francis Ga lton

Quantitative Data Analysis: Collecting data with questionnaires 09 – A questionnaire for what? The first point to consider is: ‘What kind of information do I need to collect? ’ KNOWLEDGE What people know; how well they understand something ATTITUDES What people think; what people feel BELIEFS What people think is true; how they perceive the reality BEHAVIORS What people have done; what people do ATTRIBUTES What people are; what people have The different traits require a different know-how: the more we explore the deepest sides of individuals the more becomes hard gathering useful information >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 10 – Fashioning a questionnaire from a set of questions It is crucial to think about the order in which the items will appear in the questionnaire, by taking into account the so called serial order effect Go from the general to the particular Go from factual to abstract questions Most important items in the first half of questionnaire ? ? OR ? Leave demographic and personal questions until last Another important aspect is how to organise the different sections of the questionnaire, by taking into account the so called semantic order effect Items should flow coherently, which usually requires that items on related topics should be grouped together. Coherent grouping can facilitate respondents’ cognitive processing >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 11 – How to represent the structure With a flow chart it is possible to represent the questionnaire and test the sequence of questions or sections, by using plan figures for actions and arcs for the information flow begin/end interview screener or filter question or section direction NO condition of women in employment NO stop NO labor force status start gender fem. ? YES age < 15 ? YES >>> Nottingham Business School – 2013 <<< stop emp. ? YES

Quantitative Data Analysis: Collecting data with questionnaires 12 – Question content: phrasing and wording Each question stimulate a certain response in the interviewed person: the questions should be formulated to produce an equal response in similar subjects The results may be influenced by factors such as the succession of the different options, the rate at which the interview is conducted, the personality of the interviewer… It is better to use short sentences, without many clauses and specification The questions must be answerable and not ambiguous, it is better to use a simple language The questions must be incentive compatible and allow people to provide ‘honest’ replies >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 13 – Words (don't come easy) 1. avoid confusing or unfamiliar terms 2. avoid jargon and colloquialisms 3. avoid double-barreled questions 4. avoid abbreviations or acronyms 5. avoid double negatives 6. avoid offensive terms 7. avoid foreign terms 8. avoid making assumptions 9. avoid ‘ever/never’ or ‘always’ 10. avoid questions with too many options Pay attention to leading questions or suggestive interrogations: People sometimes answer questions in a way they perceive to be socially acceptable. Make easy for respondents to admit social lapses by wording questions carefully Pay attention to sensitive topics: It is difficult to obtain information about people’s private life. To encourage responses to embarrassing questions, explain why you need to know that trait >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 14 – How wording affects responses (H. Schuman & S. Presser, 1981) A sample has been randomly divided into two subgroups of 409 and 432 respondents, respectively The researchers have asked the following question to the first group: Do you think the United States should forbid public speeches in favor of communism? The question on the same topic has been asked in this way to the second group: Do you think the United States should allow public speeches in favor of communism? Question 1 YES (39. 3%) NO (60. 7%) Question 2 YES (43. 7%) NO (56. 3%) [there is a difference of 17%] The use of two opposite verbs like ‘to forbid’ and ‘to allow’ produces an asymmetric response >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 15 – How wording affects responses (Toronto Area Survey, 1991) Question 1: In your opinion, should Sunday shopping be allowed in Ontario; that is, should stores that want to stay open on Sunday be allowed to stay open on Sundays if they want to? Results: 73% 25% 2% In favor of Sunday shopping Opposed to Sunday shopping No opinion Question 2: In your opinion, should a Sunday pause day be adopted in Ontario; that is, should the government make Sunday the one uniform day a week when most people do not have to work? Results: 50% 44% 6% Opposed to a Sunday pause day In favor of a Sunday pause day No opinion A possible explanation for the difference in the results could be that some respondents did not understand the implications of the questions >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 16 – Some response biases The trustworthiness of the answers must be assessed on the basis of possible biasing factors social desiderability - impression management - self-deceptive enhancement It is the socially shared assessment that in some cultures may be given to an attitude or a behavior acquiescence - yea saying - friendiness effect It is essentially a bias created whereby respondents tend to agree or answer positively to everything non-attitudes People may have no viewpoint towards some issues. The risk is to press people in offering opinions they actually don’t have, or induce neutral choices (‘Don’t know’ options) >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 17 – 1970 DC Comics Reader Survey Have readers/fans been ‘honest’ with DC Comics? Is it useful to encourage the respondents with gifts or monetary incentives? The problem is already debated in literature >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 18 – Don’t make things awkward for your respondents! To avoid influences on sensitive issues or reduce the social desirability, it is possible to use several tricks for reducing biases in the responses (1) indirect questions (2) self-completition (3) randomised questions Draw a card from a pack and put it back without showing: if it was a court card reply to question 1 else to question 2 1 – Have you been intoxicated by alchool in the last week? (YES NO) 2 – Does your mobile number finish with an odd figure? (YES NO) λ = 0. 45 (e. g. , percentage of YES) γ ≅ 0. 5 (probability of having an odd figure) λ = Pπ + (1 -P)γ -> P ≅ 0. 23 (probability of drawing a court card) π = ? (estimated percentage of intoxicated) 0. 45 = 0. 23π + 0. 77· 0. 5 -> π = (0. 45 – 0. 77· 0. 5)/0. 23 ≅ 0. 28 >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 19 – Different formats for different viewpoints One of the most significant decisions that can affect how people answer, is how questions are posed and which kind of information are elicited Open question Closed question information + - dispersion + - measure errors + - Strenghts Which one issue mattered most to you in deciding how to vote in the 2008 Presidential Election? Open Closed Weaknesses Open question ü ü ü Encourage free thought Solicit freedom of expression Probe for more details ü ü ü Replies can be misinterpretated May put some people off May discourage less literates Closed question ü ü ü Encourage mindless replies Produce uniform answers Easy to code and analyse ü ü Exclude the unlisted options May force to be ‘dishonest’ >>> Nottingham Business School – 2013 <<< The different types may cohabit in a questionnaire, depending on the nature of the research…

Quantitative Data Analysis: Collecting data with questionnaires 20 – Open-ended questions With open-ended question it is possible to collect both quantitative or qualitative information - Numeric open-ended It is appropriate when respondents are asked to supply a specific answer and a large number of responses are possible or when all the possible answers are not known How many exams of Statistics have you done in your previous university career? _____ - Text open-ended (Verbatims) It is appropiate when we are interested in opinions, feelings… There are several formats: Unstructured question Word association Sentence completition Story completition Thematic apperception test … >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 21 – Closed-ended questions (1) - Two-option (dichotomous) This is the simplest format. Options may include: No/Yes, Disagree/Agree, False/True… Do you remove the clippings from your lawn after mowing? No Yes - One best choice It can be used to solicit information or to test knowledge. It is appropriate when all relevant choices are known and listed. Options are supposed to be independent and mutually exclusive What does the word ‘nutrition’ mean to you? (Circle one number) 1 Getting enough vitamins 2 The food you eat and how your body uses it 3 Having to eat foods I don’t like 4 Having good health - Multiple choice This format is actually a series of ‘yes’ or ‘no’ items. It is a fast and easy way to obtain information and also save space. But… don’t make the list too long or the respondents may not consider each item! >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 22 – Closed-ended questions (2) - Ranking Rank ordering is a multiple choice. Respondents are given various responses and asked to rank them in order of importance or indicate the bests x options What would you like to know more about? (Select three responses and rank them in order of importance) 1 ___ a) What to eat to look better d) Health foods 2 ___ b) How food affects you e) Physical conditioning through diet 3 ___ c ) Weight control f ) Vitamins - Rating Respondents are asked to indicate their choice at the most appropriate point on a scale. The size of the scale depends on the amount of possible and desirable differentiation, people’s capacity to answer When purchasing a new car, how important are the following traits in your selection process? (Circle one number for each trait) Selection Trait of little importance a) Performance b) Comfort c ) Safety d) Brand 1 1 2 2 3 3 4 4 5 5 6 6 >>> Nottingham Business School – 2013 <<< highly important 7 7

Quantitative Data Analysis: Collecting data with questionnaires 23 – Frequency questions Perhaps one the most frequent mistakes relate to frequency questions. Consider this question: How often do you go to the cinema? Never Rarely Sometimes Often What do the scale points of Rarely, Sometimes and Often mean? One’s sometimes may be another’s often. When measuring frequency it is always important to give people precision about time: Thinking about the last twelve months, how many times did you go to the cinema? None Once or twice 3 to 6 times 7 or more times Time interval has to be related with the nature of the surveyed activity: people may not be able to answer accurately if the interval is too wide, but with sporadic events is better to use larger intervals >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 24 – Measuring attitudes: the Likert scale The Likert scale (or ‘summative’ scale) is often used in Psycology as well as in Market Research for measuring people’s attitudes and behaviours It uses answer choices that range from one extreme to another: unlike a humble ‘yes/no’, a Likert scale allows to uncover degrees of satisfaction, effectiveness, likelihood, and so on Rensis Likert question matrix Each option has a given score (e. g. , 1 2 3 4 5 ): the total (or the average) score represents the respondent’s position on the scale at wh e som dis in e y l e r rta g ng e a o c s r St Di Un at ee wh e Agr m o y l s ng ee Agr Stro ITEM 1 1 2 3 4 5 ITEM 2 1 2 3 4 5 ITEM 3 1 2 3 4 5 ee agr >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 25 – Discriminating power of an item Discriminating item Non discriminating item 50 50 40 40 30 30 20 20 10 10 0 stro ngly unfa unf vora ble avo rabl und ecid ed favo rabl e 0 stro ngly fav e orab ngly le Discriminating item underlying a negative value 50 45 40 35 30 25 20 15 10 5 0 stro und unfa ecid vora ed ble avo rabl e unf favo rabl e stro ngly fav orab le Discriminating item underlying a conflict 50 40 30 20 10 stro ngly unfa unf vora avo rabl ble e und ecid ed favo rabl e stro n gly f 0 avo rabl e stro n und unfa ecid vora ed ble nfav orab le gly u >>> Nottingham Business School – 2013 <<< favo rabl e stro ngly fav orab le

Quantitative Data Analysis: Collecting data with questionnaires 26 – Some format guidelines Questions should have the following characteristics: 1 a) a number b) a common format c) clear instructions d) appropriate space for ‘write in’ answers e) a format that distinguishes Don’t know, No opinion and Not applicable options from lists of scale answer options It would be appropriate to consider partially closed-ended questions if we are not sure about the answer options: the most frequent choices can be counted analysed NO 2 It is better to not split questions across different pages! >>> Nottingham Business School – 2013 <<< YES

Quantitative Data Analysis: Collecting data with questionnaires 27 – General advices Maximize your response rate by providing clear information and instructions as follows: 1) State ever who you are 2) Outline what the purpose of the survey is and why their response is important 3) Explain how answers will be treated with confidentiality and anonymity 4) Provide clear instructions on how each question should be answered It is really important to take under control how much time requires completing the interview 15’ – 20’ Face to face interview 10’ – 15’ Self-completition 05’ – 10’ Telephone interview >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 28 – Wrap up your questionnaire The appearance of your questionnaire will go a long way towards encouraging responses Use different colours and images to make it attractive Provide plenty of white space between questions so the questionnaire doesn’t look too ‘busy’ Use clear headings if appropriate Although it’s tempting to use smaller fonts in order to squeeze your questionnaire onto a smaller number of pages, don’t make it so small that it becomes illegible >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 29 – Before going on the field… No matter how closely a questionnaire follows recommendations and best practices, there is no substitute for testing it Does each question measure what it is intended to measure? Do respondents understand all the words? Are questions interpreted similarly by all respondents? Does each question have an answer that applies to each respondent? Does the questionnaire motivates people to answer it? Ask colleagues to critically review the questionnaire (peer review) and make a ‘final rehearsal’ by piloting the survey on a small sample. After debriefing you are ready to start! >>> Nottingham Business School – 2013 <<<

Quantitative Data Analysis: Collecting data with questionnaires 30 – Test yourselves Design a questionnaire for interviewing the potential users for the launching new social network CARbook! The questionnaire would have to contain at least 10 questions about the users’ preferences Test the questionnaire on your master colleagues and statistically describe the data you have obtained >>> Nottingham Business School – 2013 <<<