COLLECT ADDITIONAL DATA The Case for Qualitative Data

COLLECT ADDITIONAL DATA The Case for Qualitative Data Collection, Study Stages, & Protocol Development Day 1 - Session 3

Session Objectives § Distinguish between quantitative and qualitative data collection § Identify the stages of a qualitative investigation/study § Describe the components of an investigation/study

When to use qualitative research § When events are unknown or need to be explored § When events change quickly § When we lack good ways to measure things § When we need to know what real life is like § When the numbers and statistics do not “add up”

When not to use qualitative research § Is not good when: § Understanding is already complete § To test hypotheses that are fully bought into § To generate measures of statistical significance and generalize to other contexts

Useful Distinctions between Qualitative and Quantitative Research Qualitative: Inductive Primary data Exploratory Interview/ Observation Small sample No systematic sample Words Source: Randall Teal, UNC the North Carolina Translational & Clinical Sciences Institute Quantitative: Deductive Secondary data Confirmatory survey Large sample Systematic/Random numbers Viewed more as a continuum than a separation

Quantitative vs. Qualitative § Quantitative (numbers) − Answers the how much − Useful for determining the size and scope of problem − Often used to prove the existence of relationships § Qualitative (words) − Answers the how, what, & why questions − Used to generate understanding of issues from the perspective and in the language of the informants − Ideally suited to explore sensitive topics in depth with small numbers of informants

Qualitative Research For public health administrators and policy makers, qualitative research is an important tool to study the ‘‘black box’’ or to understand the processes and factors that contribute to the success or failure of interventions or policies.

Qualitative Research Methods Interviews Participant/ Non-participant observation Focus groups Document Review

Key Informant Interviews Ø Same learning goals as other methods Ø Structured, semi-structured Ø Can interview among different participant groups Ø Useful for sensitive topics Ø Can explore responses in depth

Observation • Types of observation Structured, Unstructured, Participant • Taking in everyday life • Jotting notes • Writing up field notes

Focus Groups § Participants 6 -10 people per group Common characteristics Shared experiences Compare different groups § Environment Comfortable and familiar (snacks/beverages always a plus) Circle seating (if possible) 2 hr session maximum Tape recorded if possible (note-taker is a MUST)

Focus Groups When to use focus groups § Group interaction desired § Want range of ideas § Group differences § Discuss 1 -2 issues When NOT to use focus groups § Free expression cannot be ensured § Confidentiality cannot be protected § Participant(s) have problems with social aspect of being in a group

Document Review • Researcher works with whatever is at hand • Documents provide a window on past events or ideas • Often combined with other methods

Which Method? Method Best Use Interviews Sensitive information, require in-depth responses from each interviewee. Observations Want to observe actual setting with objective “checklist, ” or participate in others’ experiences Focus Group Non-sensitive information, limited time, range of view on 1 -2 topics. Document Review Low budget, existing materials with information available.

Data Collection Process 1. Develop a Concept Note 2. Planning 3. Implementation 4. Data Management and Analysis 5. Planning and Assessing Data Use

Data Collection Process 1. Develop a Concept Note § What question you are going to answer § What data collection instruments you are going to use § Who you are going to collect information from § How you will analyze the data § Protocol? ?

Concept Note Elements A. Rationale B. Study purpose (objectives) and questions C. Participant selection D. Data collection instruments E. Analysis methods F. Ethical review

Conceptual Stage - Questions § Variance § Process o “Does” o “How do” o “How much” o “In what way does” o “To what extent” o “Is there a relationship” 1) Meaning of events or activities; 2) Process by which they and associated outcomes occur; 3) Influence of physical and social context.

Example - CSI Study Purpose B § To explore how care decisions are made by community-based volunteers/social workers, and the utility of the CSI at the community level as a job aid. This information will assist MEASURE Evaluation in improving M&E guidance for programs working with vulnerable children and help us to understand more about the community caregivers providing these services to vulnerable children.

Example - CSI Study Questions B § How do community-level volunteers/workers make care decisions? What is the difference in how decisions are made between CSI users and non-users? § Among community-level volunteers/workers who report implementing the CSI, to what extent do they believe that the CSI process/data supports improved decision making? § Among communities where community-level volunteers/workers are working, to what extent are care decisions made independently by volunteers/workers versus in care teams? § Among community-level volunteers/workers who report implementing the CSI, to what extent have they been trained to use the CSI? Are there differences in the decision-making process based on characteristics of volunteers/workers?

Activity 2 – Case Example (Part 1) § In your group, read the handout (10 minutes) § Answer the questions (45 minutes) § Present findings (20 minutes)

Activity 2 – Case Example (Part 2) § Write your study purpose and 2 study questions on flipchart (40 mins) § Present to plenary (20 mins)

Participation Selection C § Where to collect data? § From whom to collect data? § Parents, Clients, Program Participants, Staff § At times, may want to collect data from more than one group.

Conceptual Stage - Sampling § “Theoretical” versus “A Priori” § Types of “A Priori” sampling: § Extreme § Intensity § Homogenous § Heterogeneous § Typical cases § Snowball § Opportunistic C

Participant Selection § No formula § Goal is data “saturation” § Depends on # of topics & sub-groups § Many topics, then many informants § More depth, maybe less informants § Identify criteria § Assess resources § Typical range 20 – 120 informants C

Activity 2 – Case Example (Part 3) § Based on your study purpose and questions, discuss where you will collect data [level] and from whom [who can best answer your questions]? (15 mins) § How will you select your participants? How many will you select? Why? (15 mins) § Transfer where, who, how, and how many participants on to flipchart paper (5 mins) § Present to plenary (25 mins)

Feedback & Adjourn

The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00 -08 -0000300. Views expressed are not necessarily those of PEPFAR, USAID or the United States government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc. , Management Sciences for Health, and Tulane University.

www. measureevaluation. org

Complementing Routine Data with Qualitative Data for Decision Making: Understanding the “Why” Behind Program Data Day 2

Day 1 Review § When to use qualitative methods § Qualitative methods § Data collection process § Start with a concept note

Concept Note Elements A. Rationale B. Study purpose (objectives) and questions C. Participant selection D. Data collection instruments E. Analysis methods F. Ethical review

Data Collection Instruments § Key elements for section: § Entry strategy § Who will collect data § How will data be collected § Where data will be collected § What data will be collected D

Interview Questions D § Study questions: § Outline broad association that you are interested in exploring § Identify what you want to understand (or purpose) § Interview questions: § Detail specific aspects of associations you want to explore in a format appropriate to ask informants § Generate the data needed to answer the research question

Creating interview questions D § These are the questions you will ask informants to help you answer your research questions § Think about: § Language § Breadth and depth of question § Respondent burden § Logical sequence of questions

Types of Interview Questions D § Opinion: What do you think about a girl at this age getting pregnant? § Feeling: How did you feel when you learned you were HIV-infected? § Knowledge: Tell me what you know about health services for pregnant girls in your community?

Types of Interview Questions § D Grand Tour: tour through space, time, events, or people, activity, or group of objects Ø Typical: Can you describe a typical day in this clinic? Ø Specific: Can you tell me what happened when you came to work in this clinic yesterday from the moment you arrived until you left? Ø Task: Draw a map of the clinic. § Mini Tour: What happens in pre-test counseling sessions with clients you see in this clinic?

Types of Interview Questions D § Example: Could you give me an example of how you recently counseled an HIV+ client to share his/her HIV test results? § Experience: Can you tell me about your experiences receiving services at this clinic?

Types of Interview Questions D § Native Language: Ø Direct Language: How would you refer to it? Ø Hypothetical-Interaction: If a married, female client were to walk into your office today for an HIV test, what kind of things do you think she would tell you?

Learning to ask effective questions D § Avoid yes/no questions § Avoid other close-ended questions § Avoid “why” questions § Avoid leading questions § Learn how to quickly rephrase questions during the interview § Understand the purpose of your data collection

Closed-ended Questions § Can be answered with a single word or short phrase. Closed questions give: § Facts § Are easy and quick to answer § Interviewer keeps control of the conversation § Example § Do you like the services provided at the health center? D

Open-Ended Questions D § Require the respondent to think and reflect § Respondent will give you opinions and feelings § Respondent has control of the conversation § Open question words § What, how, where, when, who § Describe, explain, tell me about § Example: What do you think of the health services offered at the clinic?

Leading/Non-leading Questions D § Leading Questions – Worded in a way that influences the participant’s response to try to lead them in a certain way of responding § Non-leading Questions – Free of preconceptions, truly seeking the opinion, thoughts, feelings of the respondent.

Example D § Leading Question - Did they change? § Response: Some of them did. § Non-leading Question - What did you find out? How did participation in the program affect participants? § Response: Many participants reported “transformative” experiences by which they meant something life-changing. Others became more engaged in experiential education themselves. A few reported just having a good time.

Example D § Leading Question - Were you doing a formative evaluation? § Mostly § Non-Leading Question - What were the purposes of the evaluation? § First, to document what happened; then to provide feedback to staff and help them identify their ‘model’; and finally to report to funders

Group Activity D § Read the question as a group § Discuss what is the problem with the question – § Close-ended question § Leading question

Example 1 D § Actual Question - Were you trying to find out if the people changed from the economic strengthening training? § That was part of it • Better Question - What were you trying to find out through the ES training evaluation? § Several things. The types of income generation activities that participants had experience with and have had success with in the past. Also, we learned about how some income generation activities had failed and what were the reasons for that.

Example 2 D § Actual Question - Did the participants change? § Some of them did. § Better Question - What did you find out? How did the training affect participants? § Many participants reported ‘transformative’ experiences by which they meant something lifechanging. They connected with each other on different income generation activities and talked about how to move forward collaboratively on some of the ideas.

Example 3 D § Actual Question – How did this program meet your needs? § It met my needs by helping me to link to services § Better Question – How did this program address your needs? § Well, it met some of my needs – it helped me link to health services, but it didn’t meet my needs in terms of education and food-related issues.

Interview Question Sequence § Early § Inviting, easy, and interesting questions § Avoid sensitive questions § Create rapport § Main part § Most important questions § Requires thought or special effort § Closing § Some easy questions § Uncomfortable questions (high rate of non-response) D

How many interview questions? D § Limited by length of interview (30 – 45 minutes) § 5 -6 main interview questions § Each question may have 2 or 3 probes. What influences how HIV prevention services are distributed across Iringa? Probe: What information sources do providers rely on to tell them where services are needed?

Activity 2 – Case Example (Part 4) § Based on your study purpose, questions, and participants, develop one brief interview guide (no more than 7 questions) for a participant group (30 mins) § Transfer questions onto flipchart paper (10 mins) § Present to plenary (20 minutes)

Analysis Methods E § Protocols will describe the analysis methods to be used after data are collected: § How will data quality be assured § Who will do data analysis § What program will be used and the type of analysis § What is the unit of analysis (clinic vs. provider vs. community)

Study Protocol – Ethics Review F § Purpose: Protect human subjects § Who: Institutional Review Board (IRB), Human Research Ethics Committee (HREC) § http: //ohrp. cit. nih. gov/search. aspx? sty p=bsc, Click on the “IRB” tab and then “Advanced Search. ” § Important: exemption/approval required before data collection begins.

Study Protocol – Ethics Review F § Institutional Review Board (IRB) ü Confidentiality ü Data sharing ü Data security ü Informed consent ü Recruitment methods ü Privacy ü Risks ü Benefits & costs

Study Protocol – Ethics Review F § Categories: Exempt, expedited, full review § Exempt – low risk, part of people’s jobs, anonymity or confidentiality assured § Expedited – in between § Full Review – high risk categories, biological specimens.

When IRB oversight is required F Ø Collection and/or analysis of biological specimens Ø Review and/or analysis of medical records Ø Surveys or interviews with children Ø Surveys or interviews with prisoners Ø Surveys or interviews specifically with individuals whose behaviors place them at risk of criminal and/or social liability Ø Surveys or interviews which ask respondents to disclose sensitive information

Introductory Script – Informed Consent § Purpose of research § Risks § Costs § Benefits § Contact information § Option to discontinue at any time F

MEASURE Evaluation is funded by the U. S. Agency for International Development (USAID) and the U. S. President's Emergency Plan for AIDS Relief (PEPFAR). Views expressed in this presentation do not necessarily represent the views of USAID, PEPFAR or the U. S. government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc. , Management Sciences for Health, and Tulane University.