How to Design a Questionnaire Survey Moshe Banai

How to Design a Questionnaire / Survey Moshe Banai, Ph. D Moshe. banai@baruch. cuny. edu

Questionnaires Standardized, structured instrument Focus on closed ended questions Administered in a standard way

Questionnaires Best used when: There is a large sample You want fairly straightforward information You want standardized data from identical Questions You are more interested in what occurs rather than why or how

Limitations of questionnaires Can be superficial - difficult to capture the richness of meaning Cannot deal with context - information is collected in isolation of environment Information is not causal - cannot attribute cause-effect relationships Information is self-reported - which does not necessarily reflect actual behavior

Types of surveys Mail: – cheap, wide coverage, standardized, low response rate Telephone: – medium cost, wide coverage, medium response rate, standardization depends on interviewer Face to face: – most expensive, coverage depends on personal contact, highest response rate Internet: - cheapest, wide coverage, standardizes, no control over respondents , unknown response rate

Designing a questionnaire Is a questionnaire appropriate? Identify the resources that are available Decide what information you need Select items for inclusion Design the individual questions

Developing your questions Search the literature – bibliographic databases - citation searches of key articles Preliminary research – focus groups – key informants interviews

Type of information Knowledge - what people know Opinions, attitudes, beliefs, values - what people think about an issue Behavior - what people do Attributes - what are people’s characteristics Memory - based on self-report

Knowledge Example: What is the standard assignment time for expatriate managers in your company? 1 year 2 years 3 year 4 years Not sure

Opinions Example: What do you think are the major issues affecting health care in the USA at the moment?

Behavior Example: Have you developed a succession plan for any of your subordinates? Yes No

Attributes Example: When did you graduate from university?

Types of questions Open-ended questions What? Why? How? No predetermined responses given Able to answer in own words Useful exploratory research and to generate ideas Flexible Requires skill in asking questions and interpreting results Answers can lack uniformity and be difficult to analyze

Open-ended questions What do you think about the quality of the training you have received from your employer before you started your job? ________________________________

Types of questions Close-ended questions Designed to obtain predetermined responses (Yes/No; True/False; strongly agree-strongly disagree, etc. ) Easy to count and analyze Easy to interpret May not have catered for all possible answers Questions may not be relevant or important

Closed-ended questions Example: My compensation package is very similar to the standard for my job in the industry: Strongly Agree/Neutral/ Disagree/ Strongly disagree

Close-ended questions Example: Please rate the quality of the service you have received on the plane during your flight from Istanbul to Almaty: Poor Fair Good

Filter questions useful to ensure respondents only answer relevant parts of questions

Filter questions Example: Unfiltered: If you use a HR software program, which one do you use? Filtered: Do you use a HR software program? No - jump to next question Yes - which one?

Filter questions ‘Skips’ in questionnaires more easily managed if these are computer-assisted Consider including ‘not applicable’ category Example: – In the past week, how often have you used your HR computer program to check on employee’s employment status? Not at all At least once More than once I do not have access to ‘employment status’ on my computer

Getting the question right Question wording – Questions need to be clear, simple and precise – Poorly written questions lead to ambiguity and misunderstandings and can be wasteful Responses – open, closed, what type of response set will you use?

Common problems with wording Leading questions: A question phrased in a manner that tends to suggest the desired answer

Common problems with wording Leading questions: Example: Do you prefer being managed by a manager of your own sex? Would you rather be managed by a: Male manager Female manager Either/doesn’t matter

Common problems with wording Vague questions: Not clearly or explicitly stated or expressed question

Common problems with wording Vague questions: Example: In the last 12 months, have you visited your doctor? How long has it been since you last visited your doctor? within the last month between 1 and 12 months ago more than 12 months ago

Common problems with wording Biased or value-laden questions: Questions that impart a personal value of the speaker that may not be true in the strictest sense but based on personal opinion/values, especially those that attempt to be global in scope. Example: Do you think evidence-based management is a waste of time? What do you think of evidence-based management?

Common problems with wording Threatening questions: Questions perceived by the respondents to be a source of danger or menace.

Common problems with wording Threatening questions: Example: 1. What is your average number of days of drinking per month? _____ Days 2. What is the average number of drinks you consumed on each of these occasions? 1, 2, 3, 4, 5, 6, 7 and over Mooney, L. Gramling, R. 1991. “Asking Threatening Questions and Situational Framing: The Effect of Decomposing Survey Items. ” The Sociological Quarterly, 32(2): 289 -300.

Common problems with wording Threatening questions: Example: How many times per week do you drink at home? How many drinks do you consume on each of those days? How many times per week do you go to the Pub? How many drinks do you consume on each of these visits? How many times per week do you drink at work? How many drinks do you consume on each of these days? How many times per months are you invited to parties? How many drinks do you consume on each of these events? How many times per months do you attend sport events? How many drinks do you consume on each of these events? This format has resulted in a much higher level of consumption

Common problems with wording Double-barreled questions – two concepts in one question Example: To what extent does your employee work hardly and accurately? To what extent does your employee work hard? To what extent does your employee avoid mistakes in her work?

Common problems with wording Negative questions: – avoid using negative wording ‘not’, ‘rarely’, ‘never’, or words with negative prefixes ‘inefficient-’, ‘impossible-’, ‘unreasonable-’. Example: Managers should not delegate to employees they do not trust: agree / disagree Managers should delegate to employees they do not trust: Agree/ Disagree

Common problems with wording Complex questions: Questions that contains multiple categories Example: On a scale of 1 to 10, please rate for each of the 9 categories listed below, your level of knowledge, confidence and experience. Please complete the table below about your level of knowledge, confidence and experience in each of the following areas. Please complete the table below about your level of knowledge in each of the following areas

Common problems with wording HR Activities Recruitment Selection Orientation Training Career Planning Evaluation Compensation Benefits Global Knowledge Confidence Experience

Responses Closed ended questions are usually followed by a set of responses Choose type of scale: Nominal Ordinal Continuous (summed items with ordinal response scale) Interval Ratio

Responses: Nominal Data are neither measured nor ordered but subjects are merely allocated to distinct categories Example: Are you: 1. Male 2. Female What is your marital status: Single Married Widowed Divorced Separated Other

Responses: Nominal • Limited choices of responses, lack of consistency in what a yes/no, agree/disagree response means Example: Do you have trouble training your employees? • Attitudes and behaviors lie on a continuum To what extent do you experience difficulty when training your employees? None A little Quite a bit A lot I do not train employees

Responses: Ordinal Data about the rank order of scores Example: What is the highest level of education you have reached: Did not complete primary school Completed primary school Up to, but not including year 10 Completed year 10 or equivalent HSD or equivalent Associate Degree BA or equivalent Masters Ph. D or equivalent (JD, DBA, etc. )

Responses: Ordinal Continuous Types of ordinal continuous response scales – Visual analogue scales Example: Overall, how much satisfaction do you get from your job? Low satisfaction _V_______ High satisfaction

Responses: Ordinal Continuous Provide adjectives for points along the line (adjectival scales) I No Sat I little Sat I Some Sat I high Sat I very high Sat I

Responses: Osgood’s Semantic differential Data about connotative meaning of concepts where scale lies between two bipolar adjectives Example: I am: Shy ___________ Gregarious Serious ___________ Funny Sad ____________ Happy Thinking___________ Feeling Reactive____________ proactive

Responses: Likert scale – rate agreement with a series of statements Example: To what extent do you agree or disagree with each of the following statements: Strongly Agree Neither Disagree Strongly disagree

Responses: Likert scale How many steps/boxes should there be? – five to seven response categories ideal People averse to extreme ends of a scale – avoid absolutes; almost always vs always, almost never vs never – add throw away categories at either end

Responses: Likert scale Should there be an even or odd number of categories – not an issue if your scale goes from ‘not at all’ to ‘very much’ (unipolar scales) – If your scale is bipolar (eg: strongly agree to strongly disagree), decide whether you want a ‘neutral’ point

Scales’ structure, statistics and transformation Scale Type Mathematical structure Permissible Statistics Admissible Scale Transformation Nominal categorical Standard set structure (unordered) Mode, Chi-Squared One to One (equality(=)) Ordinal Totally ordered set Median, percentile Monotonic increasing (order(<)) Affine line Mean, standard, deviation, correlation, analysis of variance Positive linear (affine) One-dimensional vector space All statistics permitted for interval scales plus the following: Geometric mean, harmonic mean, Coefficient of variation, logarithm Positive similarities (multiplication) Interval Ratio

Problems with responses Effort required to answer questions –During your last annual evaluation with your manager , did the manager discuss ways to help you progress in your career? -What is meant by discuss? - Relies on recall of discussion - Many respondents will tick a response that is satisfactory just ‘tick a box’.

Problems with responses Fatigue/boredom/disinterest – agree with everything – just say ‘don’t know’ – always choose first response – ‘randomly’ respond without considering the question Social desirability Aversion to extreme ends of the scale

Problems with responses Minimizing fatigue/boredom Keep task simple – easier to recall more recent events Keep words short and easy to understand Maintain motivation of participants – ensure task is relevant Ask people to justify their responses

Problems with responses Minimizing social desirability – is difficult – instruct that it is ok not to know something

Problems with responses Aversion to extreme ends of scale Avoid absolutes ‘never’, ‘always’ Expand number of categories by including throw away categories at the end: – never, almost never, infrequently, sometimes, usually, almost always, always

Problems with responses Minimizing ceiling effects ‘Average’ response doesn’t have to be middle response Unsatisfactory/ Average/ Above average/ Very much above average/ Outstanding

Problems with responses Halo effects – often occur when evaluating individuals – judgments made on aspects of a person’s performance influenced by overall impression of the person – a global summary just as informative

Problems with responses Framing effects Scenario 1: With training, your chance of being fired by your manager is reduced by about 34%.

Problems with responses Scenario 2: Without training, your chance of being fired by your manager is 6 out of 1000. With training, your chance of being fired by your manager is 4 out of 1000.

Problems with responses Framing outcomes in terms of ‘survival’ or ‘being fired’ will also influence responses Be careful how you ‘frame’ your questions; aim for neutral terms If unavoidable, present all relevant information

Problems with responses Order effects – May be more likely to endorse first or last response – Preceding questions may influence responses to questions that follow

Problems with responses Randomize order of response sets between individuals Randomize order of items within questions May be possible to randomize order of questions Don’t always present ‘positive’ or ‘negative’ sounding response first Easier to randomize in computer-assisted interviews than paper & pen questionnaires

Ordering questions Sequence should be logical to the respondents and flow smoothly from one question to the next Questions tend to flow from: – general to specific – impersonal to personal – easy to difficult

Validity and reliability Validity: – question measure what you claim it measures – problem with self-report Reliability: – results are reproducible or consistent with similar groups of respondents, over time and when other people administer the questionnaire

Layout Just as important as wording Aim for a professional look Tips: – cover letter/introductory page giving study title, organization, aims of the survey – enough space for open-ended questions – font large enough to read without strain – consistent and clear instructions – don’t split questions or answers across pages – enough white space

Designing a questionnaire Compose the wording Use plain language and simple questions Use mostly closed ended questions Determine layout Prepare a first draft

Designing a questionnaire Pilot – Discuss it with your colleagues – Pilot it with the target group and as you intend to administer it Evaluate and modify on basis of pilot Conduct survey, including protocol

Ethical considerations Protect the rights of free will, privacy, confidentiality and well-being of research participants, and minimize the burden of study participation to the greatest extent possible Maintain sensitivity to cultural and social differences Observe professional standards for managing and conducting scientifically-rigorous research at all stages of the study

Ethical considerations Report research findings and methods and provide appropriate access to study data Institute and follow appropriate quality control procedures Document materials and procedures related to the ethical conduct of the study and ethics committee reviews