Developing a Fieldwork Plan for Qualitative Evaluation insert

Developing a Fieldwork Plan for Qualitative Evaluation [insert date] Qualitative Methods in Evaluation of Public Health Programs Session 9

The evaluation process

Learning objectives By the end of the session, participants will be able to: 1. Understand what qualitative data collection in evaluation requires 2. Outline field data collection, identify timeline components, and find potential solutions to timing constraints 3. Describe key components of a field data collection budget and potential solutions to budget-related constraints

Learning objectives By the end of the session, participants will be able to: 4. Describe the interviewer field team: Hiring, training, and field supervising needs 5. Understand considerations related to the funding agency or government regulatory body requirements 6. Recognize the special considerations required for qualitative methods and the management of crisis during fieldwork

Stand Up! Activity 1!!!

Activity 1 The road of fieldwork Instructions: 1. Split group into teams of 4– 5 2. Each group gets cards with the steps of qualitative fieldwork 3. The group will put the cards in order and later discuss together the order assigned to the cards Time: 5 minutes for the group activity, 10 minutes to discuss steps together

Fieldwork road Proposed order 1. Talking with program implementers 4. Ethics committee (IRB) 7. Fieldwork time 2. Protocol: Evaluation design 5. Piloting tools, and preparing fieldwork 8. Introduction to local authorities 3. Contract, signature (TOR) 6. If tools changed, submit again to the IRB 9. Finding the correct informants

Fieldwork road 10. Applying tools/ techniques 13. Midpoint meetings with fieldwork team 11. Taking field notes 14. Finishing fieldwork; go back home 12. Backup information 15. Data cleaning 16. Starting the “formal” analysis time

Activity discussion § What interesting conversations emerged during discussion about the exercise? § Are there any additional steps to add to the fieldwork road?

Fieldwork Time, budget, scope

Time and budget: “Ideal” world Before fieldwork § Protocol development: 1– 3 months § Stakeholder engagement § Tools: Instruments, laptops, audio recorders o Instruments (focus group discussion [FGD], semistructured interview [SSI], observation guides) o Informed consent forms § Institutional review board (IRB) requirements and local agency approvals: Minimum 3 months. § Piloting (instruments and fieldwork plan) and revisions (if major, go back to IRB) § Team training: Timeframe is variable, but training tests are useful

Time and budget (1) During fieldwork Recruiting informants and applying tools: § Different tools require a specific budget and time o One focus group discussion (FGD) using snowball sampling = 1 week of work o One in-depth interview (IDI) using snowball sampling = 4 hours § Informants must be aware of the time requirements (how long our tool will take)

Time and budget (2) During fieldwork When fieldwork is extensive, set up regular meetings: § Do constant comparative analysis § Review preliminary findings § Reflect upon possible sources of bias and ways to promote trustworthiness in your fieldwork

Time total Fieldwork times can vary depending on field, theme, and subject characteristics. 1 interview: § 4 to 6 hours total recruiting in open population 2 hours to contact people, 1– 1. 5 hours to apply the interview, and 1 hour to make field notes § Maximum of two interviews each day 1 focus group: § 1 week total in open population 1 week for recruiting, and 4 hours to develop the FGD (1 hour to prepare, 1. 5 hours to develop the FGD, and 0. 5 hours to close and leave)

Time and budget (1) After fieldwork Transcribe and translate: § 1 hour interview = 4 hours typing; up to 2 days for an FGD Code (+ memos): § Interview = 1 day coding; 2 days for a focus group Analyze: § Additional time required!

Time and budget (2) After fieldwork § Synthesize (results and report) § Seek stakeholder review

Budget Qualitative tools are neither cheap nor easy. § Depending on the topic, subject, and fieldwork conditions Budget required for: ü Recruitment of informants ü Data collection ü Data transcription (time-consuming step) ü Data coding (time-consuming step and specialized personnel) ü Data analysis (time-consuming step and specialized personnel)

Time and budget (1) Under “constraints” Ways to save time and money: § Time-consuming methods vs. timeefficient methods o Example: Semistructured interviews vs. focus groups—assuming both fit your evaluation questions § Consider conducting interviews/group discussions coinciding with existing meetings (e. g. , stakeholders already meeting for something, and adding on an FGD afterwards)

Time and budget (2) Under “constraints” Ways to save time and money: § Transcription versus analysis tables (advantages and disadvantages) § Direct translation/transcription from audio files (versus transcription in local language and then translation) § Analysis: Summarizing versus coding transcripts § Remote training (through Skype or other online platform)—still need a local principle investigator, but can have other experts join remotely

Time and budget (1) Special considerations § Confirm timelines with funding agency § Mixed-methods: Consider different timelines (and other requirements) of each research method § Different types of sampling and recruiting require different timelines

Time and budget (2) Special considerations Budget: Consider type of subjects and recruitment, incentives (if needed), financial support for informants, reimbursement of travel costs for some populations (female sex workers and transport workers)

Activity: Timeline How long does it take? April March February January World championships December Super-regional championships November State/regional championships October Qualifying season Septembe r Build and practice season August Kickoff July Pre-season June May Registration opens

Stand up! Teamwork!

Activity 2: How long? 1. Determine schedule for this evaluation: § Objective: To evaluate effects on condom use in an HIV prevention program aimed at young boys and girls (15– 18 years old) and delivered in high schools § Fieldwork to be done in two schools (one Christian, one Muslim): o 16 focus groups (4 with boys/4 with girls, 2 with first grade/2 with last grade) o 8 semistructured interviews (4 female teachers and 4 male teachers With the same cards used in the first exercise, make a timeline. All teams will start their timeline on January 1 of next year (10 mins. ) 2. Ten minutes for plenary sharing/discussion

Timeline: Small project September– October June–July April–May Fieldwork January, February, March Pre-fieldwork Transcribing data Analyzing reporting data

The fieldwork team Quality and care

Team: Key elements of data Differences between quantitative and qualitative “ideal” teams § Background or personality § Academic training § Topic, questions, types of informants o Consider race/gender/age/class issues for pairing interviewers with participants, etc. § PI/study coordinator, logistics coordinator, trainer, driver, study team members § Translators (also discussed in ethics)

Team: Training (1) Field team must have a deep understanding of: ü Research paradigm ü Theoretical framework ü ü ü (concepts/indicators) Evaluation type and objectives Intended use of the data Qualitative versus quantitative nature of data Ethical considerations in evaluation Emergency procedures

Team: Training (2) Inexperienced or poorly trained teams produce bad data. t y er d § SSI Interviews o Training + some exercises ad a at = d B § Focus groups and participatory techniques o Experienced + high specialization in the subject § Participant and non-participant observation o Training + high specialization in the subject § Note taking o Training + some exercises All of them need supervision ba ev g n hi

Team: Gender considerations Staffing and training § Male interviewers for male participants? Females for female participants? Consider the topic and local context—what will be most acceptable? § Do you have both men and women on the study team for balanced perspectives? § Depending on topic, may need gender expert on team o For example, if you have a study on a topic such as gender-based violence or LGBT stigma and discrimination, you must have someone with expertise on gender and special ethical concerns § Training should always include discussion of gender related to the topic and/or the logistics (more later)

In the field: Logistics (1) § Field teams § Supervision § Field notes § Backup data § Daily debriefs § Emotional support for sensitive topics o Example: Cases of extreme violence where you must give emotional support to subjects and fieldworkers, and maybe take action

In the field: Logistics (2) Special gender considerations Data collection § Must take into account stereotypes and social and cultural factors that may induce gender biases § Consider who/what/where/when? o Are women and men available at different times of day because of gender roles? (e. g. , women preparing meals at certain times of day, versus men working in rice fields at certain times of day) o Are there certain locations that are taboo for men versus women to go?

In the field: Safety and behavior The following aspects must be contemplated: § Fieldworkers going into homes § Working in “hot” sites (neighborhoods, bars, streets) Field behavior: § No drinking. § Consider local norms/rules, including on gender: How to dress; who to talk with Sleeping in sites during the fieldwork: § Fieldworkers sleep in the same place

Authorities and fieldwork Funding agency and government regulations

Local and national rules (1) Always ask local authorities for permission. Show identifications and permissions. § Use a picture ID, if possible § Consider approvals by authorities from the local, district, or national level § Consider time requirements when evaluating programs that require special permissions for fieldwork o Example: Ministries of health have their own institutional review board (IRB)

Local and national rules (2) § Transparent laws (to be discussed in the ethics session at a later date) § Plans to disseminate findings at the local level

Flexibility in fieldwork Funding agency: Qualitative evaluation requires flexibility § Flexibility built into the terms of reference § Changes in design (how to discuss it with funding agency, how it can change results) § Implications of notable or unintended results Working with special populations, such as: underage; drug users; lesbian, gay, bisexual, and transgender (LGBT) people; prison populations (more in ethics session) § Can take more time and have special ethical requirements

Evaluation and crisis During fieldwork

Crisis during fieldwork (1) How to respond § Preparing: Field team training o People will tell them very personal stories § Responses/support o For participants: Field team must be prepared with information on where to receive help—legal/medical/counseling o For interviewers: Daily debriefs; monitor team for signs of emotional distress; provide referrals, if needed

Crisis during fieldwork (2) How to respond § Beyond information: How to identify cases where the research team must offer something more than “information” § Considering extreme cases o Child abuse, medical negligence, sexual exploitation/human trafficking, police involvement o What are country’s legal requirements? § Reporting to IRB—bound to report “adverse effects”

So… Which tools are expensive in terms of time and budget? Which elements would you take into account when hiring a fieldwork team? Which aspects would you clarify when establishing terms of reference with a funding agency?

Summary § Fieldwork requires significant time and resources—plan in accordance with the type of qualitative tools you are using, and ensure that informants, funders, and team members are aware of requirements § Appropriate training and support of the field team is essential § Respect local and national rules, be flexible, and plan to disseminate findings at a local level when possible

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14 -00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc. ; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www. measureevaluation. org