QUANTITATIVE METHODS Data Collection 1 Quantitative Research As

QUANTITATIVE METHODS: Data Collection 1

Quantitative Research As opposed to qualitative research, quantitative research is mainly concerned with numbers and data are easily quantified. The most popular quantitative technique is the survey, often based on a large number of cases. Surveys can be administered by mail, telephone, face to face, or more recently by the Internet or World Wide Web. 2

Survey Research Surveys provide one way of obtaining and validating knowledge. Surveys are a systematic way of asking people to volunteer information about their attitudes, behaviors, opinions and beliefs. The success of survey research rests on how closely the answers that people give to survey questions matches reality that is, how people really think and act. 3

Survey Research To describe both characteristics of the respondents & the population that they represent. Information is generally collected about a fraction of the study population (a sample)--but it is collected in such a way as to be able to generalize the findings to the population. Assesses a variety of variables of interest, therefore used primarily to analyze how the variables are related. 4

Basic Type of Survey Cross-sectional surveys: ◦ Data collected from selected single sample at one point in time to represent a larger population. Example We may poll all members of a software development organization at 10 am on a particular Monday, to find out what activities they are working on that morning. This information gives us a snapshot of what is going on in the organization. 5

Basic Type of Survey Longitudinal surveys - Survey the respondent at several points in time. ◦ Trend – Different people at two or more points in time. ◦ Cohort - Study of sample population (based on certain characteristics) each time data are collected but samples studied maybe different. ◦ Panel – Data collection at various time points with the sample of respondent. A cohort may view the change in software developers activity over a day, week, month or year. Or it may track the activities of a group of testers to determine in what order different kinds of tests are administered. 6

Basic Type of Survey • Case control This study is retrospective, asking participants about their previous circumstances to help explain a current phenomenon. Suppose we are interested in whether successful designers could be identified early in their careers. We might survey our current lead designers to try to identify features common to their backgrounds, i. e. level of education and experience. 7

Survey Data Collection The type of data you collect and how you collect them affects how you can use the data. Therefore, before you start data collection you must identify ◦ Objectives ◦ Instrument evaluation ◦ Sampling ◦ Mode of data collection 8

Setting Specific, Measurable Objectives • A statement of survey’s expected outcome. • Examples –To identify the most useful features of a user interface of a system. –To identify the most common training needs for a new hires. • It is important that the objectives statement include definitions of all potentially ambiguous terms. • Clear and measureable 9

Setting specific, Measurable Objectives • Objective derived from a perceived need. –Someone in a organization wants to know something about the org, the state of practice, the effectiveness of a new technology, or the experience base of its members. • Objective maybe determine by the literature search –To find out who is writing about which new technologies and where there are gaps in current knowledge 10

Setting specific, Measurable Objectives • Expert can also provide objective –Researches planning a survey about software testing technologies may consult testing expert to determine what information is needed about the technologies. 11

Survey Instrument. Questionnaire Preparing the data collection instrument ◦ Survey instrument, usually questionnaires. Are constructed using the following steps: ◦ Search the literature. ◦ Construct ◦ Evaluate ◦ Document 12

Survey Instrument. Questionnaire The first problem that a survey researcher has to tackle is how to design the questionnaire so that it gets the right information. –Is this survey necessary? –Is the purpose of the survey to evaluate people or programs? –Can the data be obtained by other means? –What level of detail is required? 13

Survey Instrument. Questionnaire The second problem is how accurate does the survey have to be? –Is this a one-time survey or can the researcher repeat the survey on different occasions and in different settings? –How will the results be used? –How easy is it to do the survey? 14

Basic types of survey questions The way a question or statement is worded and the response options offered determine the nature of the data received. Types of survey questions include: ◦ Open-ended response ◦ Closed response ◦ Semantic differential scales ◦ Agreement and rating scales ◦ Ranking scales ◦ Checklists 15

Questionnaire Design The principles for designing a questionnaire or an interview schedule are quite straight forward but they do require careful consideration. ØDeciding what to include: operationalisation ØSelection of areas to ask questions about ØConstruction of questions ØTypes of question ØQuestionnaire layout ØIncreasing response rates

Deciding what to include: Operationalisation A first step in constructing questionaire involves establishing clear objectives of the survey. Some key questions are What is the questionnaire trying to find out? What sort of information is it required to elicit?

Lazarsfeld's scheme for measuring concepts. Paul Felix Lazarsfeld (Feb. 13, 1901 – Aug. 30, 1976) was one of the major figures in 20 th-century American Siciology. The founder of Columbia University’s Bureau for Applied Social Research. He exerted a tremendous influence over the techniques and the organization of social research.

Because Concepts and their measurement is so central to quantitative method there is a great deal of concern about the technical requirements of operationalisation. This concern revolves around two issues: RELIABILITY VALIDITY

The reliability of measures The question of reliability is to what extent does an indicator measure the same thing consistently: i. e. will people answer the same question in the same way on different occasion? Different interviewers can elicit different answers from a respondent (gender, ethnic background, dress, perceived relation to e. g. organisation, may all be influential factors affecting reliability).

The validity of measures. The question of validity is how far can we be sure that a measure really does reflect the concept to which it is supposed to be referring. This is the major problem of operationalisation. How can we be sure that the operationalised concept still measures theoretical concept? There is no objective way to accomplish this as question choice is a subjective enterprise.

RESPONDENT MUST BE ABLE TO ANSWER RELIABLY WHILE CONSTRUCTING THE QUESTIONS, RESEARCHER SHOULD ASK WHETHER THE RESPONDENTS CAN ANSWER RELIABLY E. G. HOW MANY HOURS PER DAY DO YOU USE THE INTERNET? HOW FREQUENT DO YOU USE THE INTERNET? Everyday Once in a while weekly Once in a while monthly Once in a while in a year

QUESTIONS ASKED SHOULD BE RELEVANT TO A LARGE NUMBER OF RESPONDENTS IF ASK ON ASPECTS THAT ARE NOT OF INTEREST TO MANY THE FEEDBACK RECEIVED MAY NOT BE ADEQUATE OR BE USED FOR ANALYSIS. RESPONDENT PURPOSELY THINK ANSWERS THAT ULTIMATELY MAY NOT BE ACCURATE

Avoid long questions. Shorter questions are less ambiguous. ISSUE/PROBLEM If you find that a question is difficult to frame then you may need to break it down into two or more shorter questions. APPROACH RESPONDENT CAN: • READ THE ITEMS CLEARLY AND FAST. • UNDERSTAND THE PURPOSE • CHOOSE AND ANSWER THE QUESTION EASILY

Avoid negatives Question that include the word 'not' can be difficult to understand - especially when asking a respondent to agree or disagree. E. G. AGREE DISAGREE ICT DO NOT MAKE ONE TO BE CREATIVE E. G. AGREE DISAGREE ICT DO MAKE ONE TO BE CREATIVE

AVOID BIAS THERE ARE QUESTIONS THAT ENCOURAGE RESPONDENTS TO ANSWER IN CERTAIN WAYS E. G. : DO YOU AGREE OR DISAGREE TO THE STATEMENT: “THE COUNTRY WILL BE POOR IF STAFF IN THE PUBLIC SECTOR OFTEN DEMAND FOR PAY INCREASE? MANY WILL DISAGREE

Avoid ambiguous questions. Avoid questions that may be opened to a variety of interpretations. MORE SUPPORT AID FOR THE POOR RESULT OF SIMILAR RESEARCH LESS SUPPORT WELFARE 62. 8% = STATED TOO LITTLE IS ALLOCATED FOR THE POOR 23. 1 % = STATED TOO LITTLE IS ALLOCATED FOR WELFARE

SOSIAL DESIRABILITY OF QUESTION AND ANSWER E. G. : A MALE RESPONDENT MAY AGREE THAT WOMENS’ PLACE IS IN THE KITCHEN, BUT WOULD REFRAIN FROM GIVING HIS GENUINE OPINION GUIDE: REFLEX HOW YOU WOULD FEEL WHILE ANSWERING THE QUESTION. IF YOU FEEL UNEASY, STUPID…. THEN YOU OUGHT TO THINK AND RECONSTRUCT A MORE APPROPRIATE QUESTION MANY WILL DISAGREE EVENTHOUGH THEY AGREE

Is the respondent likely to have the necessary knowledge? If you wish to ask a question about the future strategic plans of an organisation it makes more sense to ask someone who is involved in developing them. Filter devices are very useful here so that the respondent will only be directed to a particular question if he/she answered 'yes' to an earlier filter question.

CONTINGENCY QUESTION ISSUE/PROBLEM THERE ARE QUESTIONS THAT ARE ONLY RELEVANT TO CERTAIN GROUPS OF PEOPLE SOLUTION PROVIDE RELATED QUESTION TO THE EARLIER QUESTION ASKED.

FORMATING/TECHNIQUE SHOULD NOT BE UNCLUTTERED • AVOID ABBREVIATION • AVOID ASKING 2 QUESTIONS IN ONE SENTENCE. • PROVIDE SPACE /BOX FOR RESPONDENCE TO ANSWER • LIST POSSIBLE ANSWERS. • ARRANGE WELL • STATE CLEARLY THE INSTRUCTIONS TP BE FOLLOWED. IF NOT FORMATED SYSTEMATICALLY AND CLEARLY, RESPONDENTS MAY NOT ANSWER /OR LEAVE SOME QUESTIONS- INCOMPLETE.

ORDER OF QUESTIONS • ASK DEMOGRAPHIC QUESTION FIRST • ASK OPEN ENDED QUESTIONS FIRST • VARY TYPES OF QUESTION IN YOUR PRETEST • FORMAT OF SURVEY QUESTIONAIRE DIFFERS FROM INTERVIEW SCHEDULE • ASK INTERESTING QUESTIONS FIRST, THOSE THAT ARE EASY TO ANSWER

INSTRUCTIONS • MUST BE CLEAR • INCLUDE INTRODUCTION • INFORM IN EACH SECTION WHAT THE RESPONDENTS NEED TO DO. • STATE IF THE RESPONDENTS IS REQUIRED TO ANSWER ONLY ONCE. • STATE IF RESPONDENTS HAVE TO ANSWER IN RANK ORDER IS

CONCLUSION • EXAMINE PURPOSE OF THE QUESTION • CONSTRUCT ITEM WHICH ARE GOING TO BE USESFUL FOR THE RESEARCH • IF UNSURE OF THE BEST QUESTIONS TO ASK CAN ASK SIMILAR QUESTION MORE THAN ONCE. ALL THE GUIDELINES GIVEN IN DESIGNING THE QUESTIONAIRES ARE IMPORTANT TO ENSURE THAT THE RESEARCHER COLLECT THE RELEVANT DATA FOR ANALYSIS / INTERPRETATION

Questionnaire Evaluation Creating a set of questions is only the start of instrument construction. It is essential that we evaluate or pretesting it. Goal: To check that the questions are understandable. To assess the likely response rate and the effectiveness of the follow-up procedures. To evaluate the reliability and validity of the instrument. To ensure that our data analysis techniques match our expected responses. 35

Understandable Questionnaire To check whether the questionnaire is understandable: two most common ways use focus groups and pilot studies. • Focus groups. We assemble a group of people representing either those who will use the results of the survey or those who will be asked to complete the survey (or perhaps a mixture of the two groups). • The group members are asked to fill in the questionnaire and to identify any potential problems; identify missing or unnecessary questions, ambiguous questions and instructions. • Focus groups also contribute to the evaluation of instrument validity- content validity. 36

Reliability and validity In order for information to be useful, it has to be: ◦ consistent, ◦ dependable, ◦ accurate and, ◦ most of all, true. Too often, we are presented with information that fails on one or more of these criteria. In research, these criteria are represented by the concepts of reliability and validity. 37

Reliability Expect to obtain the same information time after time. If we administer it many times we will get roughly the same distribution of the result. • Test-Retest Reliability – Obtained by administering the same test on two or more successive occasions and then correlating the scores. – If the correlation between the first set of answers and: the second is greater than 0. 7, we can assume that test-retest reliability is good. – Problem is respondents remember what they said previously, so they answer the same way in an effort to be consistent (even if new information in the intervening time makes a second, different answer more correct. ) 38

Validity To make sure that our survey instrument is measuring what we want it to measure. • Face validity – Face validity is a cursory review of items by untrained judges. It hardly counts as a measure of validity at all. because it is so subjective and ill-defined. • Content Validity – It typically involves a systematic review of the survey's contents to ensure that it includes everything it should and nothing that it shouldn't. The focus group should include subject domain experts as well as members of the target population. 39

Validity Criterion validity q Concurrent criterion validity q compares a new instrument against one that is considered a "gold standard. " We can use correlations to indicate the extent to which the two questionnaires agreed. q q Can use correlations to indicate the extent to which the two Questionnaires agreed. Example: A proposed new Capability maturity model questionnaire might be compared with the existing one. q Predictive criterion validity q q Assesses the ability of a survey to predict future phenomena. Example: we can survey project managers about on-going projects and their characteristics in order to predict how much effort they will require. 40

Validity Construct validity ◦ Construct is a theoretical dimension – concerns how an instrument “behaves” when used. ◦ E. g. “ what is this instrument really measuring? ” “Does it validly measure the abstract concept of interest? ” ◦ It is essential in hypothesis testing linked to a theoretical perspective. ◦ Use factor analysis 41

Sampling We all draw conclusions about phenomena based on exposure to a limited part of the phenomena. � E. g. Patients may generalize about the friendliness of nurses at a particular hospital, based on care they received from a small number (sample) of nurses. Researchers obtain data from samples � E. g. Testing a new device/gadget � Test on a sample � Cannot administer to all production. Quantitative researchers desire samples that will allow them to achieve statistical conclusion validity and to generalize their results (adequate samples) 42

Sampling • To obtain a sample, begin by defining a target population. • Target population is the group or individuals who are in position to answer the questions and to whom the results of the survey apply. • A valid sample is a representative subsets of the target population. • If the sample is not representative, we cannot generalize our results to the target population. • Large population can be cost prohibitive • Thus researchers develop a sampling plan that specifies in advance how participants are to be selected and how many to include. 43

Sampling Sampling: The process of selecting a portion of the population to represent the entire population Sample: A subset of a population There are two types of sampling designs in quantitative research: probability sampling and non probability sampling. 44

Probabilistic Sampling Methods If we want to make strong inferences to the target population, we need a probabilistic sampling method (PSM). ◦ PSM is one in which every member of a target population has a known, non-zero probability of being selected. ◦ The aim is to eliminate subjectivity and obtain a sample that is both unbiased and representative of target population. ◦ To make statistical inferences (hypothesis testing). 45

Probabilistic Sampling Methods Simple random: ◦ Equal probability-use random generator, select the first n members on the list where n is the required sample size. ◦ Each individual is chosen randomly and each member of the population has an equal chance of being included in the sample. 46

Probabilistic Sampling Methods Stratified random: ◦ The population is divided into homogeneous subgroups/strata from which elements are selected at random ◦ A stratified sample is obtained by taking samples from each stratum or sub-group. ◦ The aim of this sampling method is to enhance representativeness ◦ Suppose a researcher wishes to study COIT students attitudes toward programming course. He could divide up students into the four sub-groups of programmes for the 1 st stratum and gender for the 2 nd stratum and take samples from these. 47

Probabilistic Sampling Methods Systematic Random Sampling Systematic sampling, sometimes called interval sampling, means that there is a gap, or interval, between each selection. Often used in industry, where an item is selected for testing from a production line (say, every fifteen minutes) to ensure that machines and equipment are working to specification. Example: the manufacturer might decide to select every 20 th item on a production line to test for defects and quality. This technique requires the first item to be selected at random as a starting point for testing and, thereafter, every 20 th item is chosen. 48

Probabilistic Sampling Methods Systematic Random Sampling In surveys, eg market researcher selecting every 10 th person who enters a particular store, after selecting a person at random as a starting point; interviewing occupants of every 5 th house in a street, after selecting a house at random as a starting point. If a systematic sample of 500 students were to be carried out in a university with an enrolled population of 10, 000, the sampling interval would be: I = N/n = 10, 000/500 =20 49

Probabilistic Sampling Methods Cluster-based: Belong to specific group Cluster sampling is a sampling technique where the entire population is divided into groups, or clusters, and a random sample of these clusters are selected. All observations in the selected clusters are included in the sample. Every element should have a specified (equal) chance of being selected into the final sample. Typically used when the researcher cannot get a complete list of the members of a population they wish to study but can get a complete list of groups or 'clusters' of the population Cheap, easy economical method of data collection. 50

Non-Probabilistic Sampling • Participants are selected from a population by non-random methods. • Less likely to produce accurate and representative samples of the population; results cannot be generalized to the population. • When people are hard to identify or rare • Used when it is not possible, or advisable, to use probability sampling. E. g. Limited resources, unable to identify the members of the population, exploratory research to establish whether a problem exists and others… 51

Non-Probabilistic Sampling Convenience: Available and willing Purposive/ judgemental sampling Snowball: Leverage recommendations Quota: Stratified until reach proportions 52

Non-Probabilistic Sampling Convenience: Available and willing � Using the most conveniently available people, who meet the study criteria, as study participants. ‘easy to reach’ population. � Weakest form of sampling, the risk of sampling bias is high. 53

Non-Probabilistic Sampling Purposive/ judgemental sampling Involves selecting a group of people because they have particular traits that the researcher wants to study. e. g. consumers of a particular product or service in some types of market research. 54

Non-Probabilistic Sampling Snowball: Leverage recommendations Identify a few key individuals Ask these individuals to volunteer to distribute the questionnaire to people who know and fit the traits of the desired sample. 55

Non-Probabilistic Sampling Quota: Stratified until reach proportions widely used in opinion polls and market research. Interviewers given a quota of subjects of specified type to attempt to recruit. eg. an interviewer might be told to go out and select 20 male smokers and 20 female smokers so that they could interview them about their health and smoking behaviours. 56

Sample Size In general, the larger the sample size (selected with the use of probability techniques) the better. The more heterogeneous a population is on a variety of characteristics (e. g. race, age, sexual orientation, religion) then a larger sample is needed to reflect that diversity. (Papadopoulos 2003) 57

Sample Size �Quantitative researchers strive for the largest sample possible. The larger the sample the more representative of the population it is likely to be. �Smaller samples tend to produce less accurate estimates than larger ones. �Researchers can estimate how large their samples should be to adequately test their research hypotheses through power analysis 58

Sample Size Power analysis is a statistical procedure. Statistical knowledge is needed before this procedure can be explained. Power calculations or analysis is conducted at the beginning of a study to: ◦ ensure you have enough participants to find a statistically significant result ◦ prevent unnecessarily large samples which waste time and resources 59

Modes of Survey Administration Personal (face-to-face) Telephone Mail Web Combination of methods 60

How do you decide on the mode of data collection? ◦ Population ◦ Characteristics of The Sample ◦ Types of Questions ◦ Response Rate ◦ Cost ◦ Time 61

Data Collection • Survey should include: – Cover letter • On letterhead • Signed by most significant person possible • Brief explanation of – Purpose – How results will be used – Why respondent was selected – Why answers are important – Emphasize confidentiality (if applicable) – Provide contact information for questions – Describe/explain any incentives for participation or consequences or not for non-participation 62

Monitor survey returns Open surveys as they are returned Assign unique ID number (if not already assigned) Stamp date on survey Track number returns daily ◦ Used to inform timing of follow-up reminders 63

Types of data § Categorical data: numbers or words are used to group things, when the answers are categories, and each respondent must fall exactly one of them. – Examples: gender, race, religion, food group, or place of residence. § Ordinal data: When the numbers are used to order a list of things – The ranking of football or basketball teams is done using ordinal numbers. – A list of things to do. – Likert scale: to assess a person’s feelings about something. § Interval and Ratio data. – Scale or interval data would be things like temperature, height, weight, age. 64

Data Coding – Coding: • Statistical analyses requires numbers. • The process of converting information into a quantifiable format (usually numerical) so that a systematic analysis of the information can be done. • The coding process is made easier by precoding responses and enabling circled or checked numbers to be entered directly into a data set. • The coding system used should model or be easily convertible to the format required by the computers software. – If the information is categorical in nature, then, the choice of whether to represent a data value as a number or a label depends upon the analyses that will be done. – If you are only going to measure frequency counts and percentages, then either a number or label is acceptable. 65

Data organization • When data is entered into a computer, it will generally be organized in a spreadsheet orientation where the rows represent different cases and the columns represent different variables. • If you leave spaces blank instead of typing in a value, you may not know if the space represents a missing value or a missing data entry. • By entering values in every space provided, you will know exactly what data has or has not been collected. 66

Resources Hulley, Cummings, Browner, Grady, Newman (2007). Designing Clinical Research, 3 rd Edition. Philadelphia. Lippincott, Williams & Wilkins. ESSENTIALS OF SURVEY RESEARCH AND ANALYSIS A WORKBOOK FOR COMMUNITY RESEARCHERS © 1998 Developed by Dr. Ronald Jay Polland for the Adolescent Pregnancy Prevention Grant, Duval County Health Department. Babbie, Earl (2002), The Basics of Social Research 2 nd ed. Wadsworth Thomson Learning: CA. Dillman, D. A. (2000), Mail and Internet Surveys 2 nd ed. Wiley: NY. Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage: CA. Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA. Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA. Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco. Tourangeau, R. , and Smith, T. W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60: 275 -304. Also see American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research. ” http: //www. aapor. org/ethics/best. html 67

Resources Aday, L. A. (1996). Designing and Conducting Health Surveys, 2 nd ed. San Francisco: Jossey-Bass. Biemer, P. , Groves, R. , Lyberg, L. , Mathiowetz, N. , & Sudman, S. (eds. ) (1991). Measurement Errors in Surveys. New York: Wiley. Dillman, D. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley. Dillman, D. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: Wiley & Sons. Fink, A. , & Kosecoff, J. (1985). How to Conduct Surveys: A Step-by-step Guide. Beverly Hills, CA: Sage, 1985. Fowler, F. J. , Jr. Survey Research Methods, 2 nd ed. Newbury Park, CA: Sage, 1993. Groves, R. (1989). Survey Errors and Survey Costs. New York: Wiley, 1989. Groves, R. , Biemer, P. , Lyberg, L. , Massey, J. , Nicholls, W. , II, & Waksberg, J. (eds. ) (1988). Telephone Survey Methodology. New York: Wiley. 68

Resources Lavrakas, P. J. (1993). Telephone Survey Methods: Sampling, Selection, and Supervision. Newbury Park, CA: Sage. Lessler, J. T. , & Kalsbeek, W. D. (1992). Nonsampling Error in Surveys. New York: Wiley. Lyberg, L. , Biemer, P. , Collins, M. , de. Leeuw, E. , Dippo, C. , Schwarz, N. , & Trewin, D. (eds. ) (1997). Survey Measurement and Process Quality. New York: Wiley. Marín, G, . & Marín, B. V. (1991). Research with Hispanic Populations. Newbury Park, CA: Sage. Turner, C. F. , & Martin, E. (eds. ) (1984). Surveying Subjective Phenomena (2 volumes). New York: Russell Sage. Journals: Public Opinion Quarterly and Journal of Official Statistics 69