Qualitative Data Analysis HandsOn insert date Qualitative Methods

Qualitative Data Analysis Hands-On [insert date] Qualitative Methods in Evaluation of Public Health Programs Session 7

The evaluation process

Section objectives By the end of the session, participants will be able to: § Design an analysis plan using a selected analytical technique § Understand main practicalities of analysis for evaluation § Demonstrate use of different qualitative analysis software and their applicability to specific analytical steps

Review analysis steps

The analysis process § What are the main stages of analysis in qualitative evaluation? § How do you construct codes and apply them? § Questions from the previous session? § Expectations for this session on analysis?

Using selected analytical techniques Designing steps for analysis

Step 1. Review the whole picture

Designing analysis Analytic strategy and analysis technique § Strategy: o Deductive or inductive (abductive) § Technique: o Content, discourse, thematic, grounded theory Source: Patton, Michael Quinn (2002)

Thematic analysis Strategy General Principle Special Case Technique Inductive reasoning Grounded theory, content, and discourse analysis Abductive Deductive reasoning

Analysis chart § An analysis chart is a tool for connecting specific objectives of the evaluation with questions, research subjects, techniques, concepts/themes/patterns (indicators) and codes § It prevents getting lost in your analysis

Analysis chart: Example Main objective/main question Here you must write main objective of the evaluation and ergo, main question to be answered. Examples are made for a formative evaluation related to the under consumption of sprinkles by under-nutrition and stunted children. Specific objectives Question to be answered for this objective Who/when/what is the source of information? Tools and topics that each tool covers Question 1. What are the strengths and weakness of the To understand the strengths program in relation and weakness of the to counseling program in relation to for sprinkles counseling for sprinkles consumption? consumption. Program operators Tool: Non participant observation Concepts: Counseling, learning strategies, of counseling barriers perception and barriers overcoming § Topic 1. Counseling § Code: Counseling knowledge information and behavior § Topic 2. Counseling practical § Code: Counseling Learning by doing session § Code: Counseling consumptions barriers § Topic 3. Counseling on § Code: Counseling Empowering methods overcoming barriers to sprinkles consumption Specific objective 2. Beneficiaries Tool: Semistructured Interview with mothers § Topic 1. Perception of undernutrition and stunted on their own children § Topic 2. Perceptions on sprinkles’ effect. § Topic 3. Strengths for consumption § Topic 4. Barriers for consumption and forms for overcoming them Concepts: Health behavior related to subjective understanding of health and subjective understanding of nutritional supplements and behavior Change behavior related to power strategies derived from vulnerable population § Code: understanding of health, stunted, and under-nutrition § Code: understanding of nutritional supplements § Code: Strengths for consumption § Code: Barriers for consumption § Code: Overcoming barriers strategies As many tools as you need As many concepts as there are involved Specific objective 1. Write each specific objective. To understand the beneficiaries’ perceptions on sprinkles, as well as the strengths and barriers for consumption. …as many lines as specific objectives there are. Question 2. Which are the perceptions of beneficiaries on sprinkles and which are the strengths and barriers for its consumption? As many questions As many persons as the evaluation as you need to must answer talk Concepts: Data will be analyzed in a dialogue with the preexistent concepts Preexistent codes

Examples What type of qualitative data do we have? Type of data Sources of data Data collector Transcripts, notes, summaries Beneficiaries, key informants Program staff Audio, video, photographs, art Program operators Evaluators and community members Diaries/notes Participants Evaluators Participatory Community leaders Evaluators mapping or timelines

Organizing data Types of data § § Field notes Transcriptions Observational notes Other data formats (video, photo, maps, art, etc. ) Families of data § Pre- and post- data § Subject type (beneficiaries and program operators) § Tool type (focus groups and semistructured interviews) § Other (region, branch of intervention, early versus late users)

Finding gaps and emergent data § Gaps: actual recruitment doesn’t match targets in sampling § Emergent: identify the need to include additional participants Revisiting the method and, as a consequence, revisiting evaluation scope/answers

Example: Finding gaps in a multicenter evaluation. Sample from “Opportunities Program” – Designed and Real

“Opportunities” national evaluation Maximum variation sampling Relevant categories 1. Program: Intervention/control 2. Structural: Community and culture § Country region (northern/southern) o Micro-region (cultural diversity) § Ethnicity (indigenous/Creole) § Household: Birth order o Firstborns (at the moment of enrollment) o Lastborns Source: Teddlie (2007); González de la Rocha Mercedes (2008)

“Opportunities” General sample Regions

“Opportunities” Sample design in detail

“Opportunities” Final sample

Revisiting the analysis plan Findings Revisiting the analysis chart: § Content, discourse, themes § Inductive and deductive (abductive) § Objective-oriented, subject-oriented, time-oriented, or theory-oriented

Step 2. Coding Figure 2. 1 Classification tree

Before coding, review themes Make sure you are on the right track When deductive strategy is used: § Themes must align with your theory of change and theoretical framework § Themes must be reflected in tools § Each theme must have enough codes

Once you are clear on themes… Concrete Inductive approach Abstract Code Category Code Themes/ concepts Code Theory Category Code Particular Deductive approach General

For example Geographical accessibility Economic accessibility Codes Theme 1 Access Acceptability Opportunity Follow-up Etc. Availability Theory Friendly services for adolescents Theme 2 Etc. Theme 3

Coding Once you are clear on themes… § Create codes § For each code, you have data § For all meaningful data, you have a code

Coding Read some of your transcripts § Deductive: Use actual transcripts to test codes § Inductive: Use data to create codes

Coding Using codes and memos There's more than one way to approach coding: § Based on theory (a priori codes; deductive analysis) § Based on fieldwork (emergent; inductive) § In vivo (while coding) § Memoing Source: Patton, Michael Quinn (2002)

Coding Hands-on (1) What to code? § Field notes, transcriptions, observational notes, images, voice recordings, audio recordings Who codes? § Individuals or teams § Ideally, coders have participated in the fieldwork § Somebody who has a deep understanding of the program, evaluation questions, and evaluation theme

Coding Hands-on (2) When to begin coding? § Coding must be done after one or two rounds of “reading” raw data and making notes How much time does it take? § Interview: Up to 8 hours for coding* § Focus group discussion: Up to 1. 5 days* § Teams can divide data or transcripts for coding tasks by subject type, by tool, by site, etc. * May vary

Coding Hands-on (3) Standardization and quality when there is a team: § “Cross-coding” (All teams code the same interview, and then they check if everybody is coding in the same way) § Regular meetings with team in order to talk about the direction of the analysis o Not recommended when mixed-methods evaluation is going on

Coding Hands-on (4) Codebook is key It must be revisited regularly. § After first transcripts are coded § Is there a code that is overused? § Is there a code (or a group of codes) with no segments coded? Notes, memos, and code relationships § Taking analytical notes or writing memos or reflexive notes during coding is key for analysis o Finding relationships between and among codes is key

Exercise: Coding

Coding exercise Materials § Codebook from Chips § In-depth interview transcript Instructions 1. Individually, read codes on gender relations and roles 2. Read transcript from interview #3, lines 9 to 229 3. Discuss with a partner at least three codes, and from which lines they were applied

Critical thinking on coding Questions § What coincidences or mismatches did you find in using the codes? § Did you code words, sentences, or paragraphs? o Why did they do so?

From “big data” to in-depth data Examples of analyzing social media with qualitative methods § Cancer survivors use social media to build networks that help them cope o Posts on social media such as Twitter or Facebook support group (if you have access to group data) o Comments during virtual events, such as a webinar for cancer survivors

From “big data” to in-depth data We can analyze Tweets, Facebook, blogs, or You. Tube content, but we may consider: ü Ethical dilemmas ü Variety of data types ü Biases of each platform

Step 3. Analyzing tools

Raw data organization: Main methods § Using qualitative analysis software § Using analysis charts (Excel or Word)

Analysis chart example Here, it is possible to see some of the most common mistakes. The coders did not transcribe what people say in first person, but in the third person. This is the correct way of registering the data.

Analysis chart (1) What? Data from: § Semistructured interviews § Observation § Documents Not from: § § Focus group discussions In-depth interviews Life histories Other techniques: theater, poetry, etc.

Analysis chart (2) Who? § Personnel who participated in fieldwork o Ideally the interviewer, observer, etc. § Someone who is trained in qualitative methods § Someone who fully understands the objectives of the evaluation

Analysis chart (3) When and how long? § Wait until finishing data collection o The evaluator must create codes for data collected § Plan ahead o Semistructured interview = half day to one day filling the chart o Observation = half day filling the chart

Analysis chart (4) How? 1. Enter sociodemographic/ context data 2. Listen to the audio 3. Stop the audio, type what is important § Transcribe verbatim 4. Make a short comment on it—the preanalysis must be related to the question(s) to be answered

Analysis chart (5) Common mistakes § Typing in third person § Capture in detail only the initial minutes of the interview § Type only those fragments that support your idea or hypothesis

Analysis chart (6) Strengths and weaknesses Strengths § Save money: Use Excel or Word charts instead of paying for transcriptions and for the software § Save time: Spend less time to capture data Weaknesses § Pre-selection leads to loss of richness of data § Requires high expertise level/knowledge of topic § Not inductive, only deductive o Mainly thematic analysis

Analysis tools: Software § Software assists in analysis, but does not analyze data § Basic functions o Data storage, systematic coding, creating memos, facilitate comparison, and linking o Useful, especially when there is a large volume of data

Examples of software programs Qualitative analysis § CDC free software An. SWR (Analysis Software for Word-Based Records) CDC EZ-Text (Freeware from the Centers for Disease Control and Prevention) Other software Dedoose Atlas. ti Ethnograph Hyper RESEARCH QCA (Qualitative Comparative Analysis) QSR Nvivo Text. Smart win. MAX Max QDA Note: There are many online discussion groups for learning how to use the software programs.

Advantage of using software Compared to analysis chart § It is more flexible; you can code images, text, audio, pdf files § It can be modified at any time and can be crosschecked among team members § It can start from fieldwork on, and be modified during, analysis time § It can be iterative § It can show you visual relationships among codes (mental/theoretical maps)

What is an output? Some types of qualitative software (including Atlas. ti) generate outputs § An output is a Word document that contains a number of verbatim fragments Ordered by: § Themes, codes § Families: o Gender, age, type of intervention, region, pre- and post-, among other groups o Memos can be attached to verbatim fragments

50 Pilar Torres 2016

Some practicalities on Output generation Comparing and contrasting § Pre- and post-data § Subject type (beneficiaries and program operators) § Tool type (FGD and SSI) § Other (region, branch of intervention, early vs. late users) Remember the main objective!: Answer your evaluation question; then theory-building takes second place. “To understand the program’s theory of action as it is actually practiced and implemented, is a priority” (Patton, 435).

Step 4. Using software to Generate output

Analyzing data Grouping information: § Creating families of documents/codes § Same logic as coding; team members can answer different evaluation questions/objectives Thick description: § Description comes first; interpretation comes second § We can understand the phenomenon that is studied and draw our own interpretations about meanings and significance Analysis must give a detailed response to evaluation questions.

Software demonstration

Performing basic tasks Using software 1. Coding following the “group” perspectives from the previous exercise 2. Creating memos 3. Creating outputs for further analysis

Summarizing Qualitative data analysis § The coding process will vary based on your analysis approach (thematic analysis vs. grounded theory or content analysis) § Analysis charts and software for qualitative analysis are valuable options to aid the analysis process § Coding should be an iterative process

Paper-based coding examples

Generating themes vs. codes § Codes are usually very specific § May have over 100 codes, not possible to write about all § Cluster codes together in meaningful units to create themes § Sometimes also known as code categories § Themes become the subheadings of the final chapter in your thesis or your report—they provide the structure § The codes in each theme give you a road map of the different aspects you need to cover in the discussion of each theme

Themes example (1)

Themes example (2)

References Fereday, J. , Muir-Cochrane, E. (2006). Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods; 5(1). Miles, M. B. , & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. 2 nd Ed. Thousand Oaks, CA, USA: Sage Publications, Ltd. Patton, M. Q. (2002). Qualitative research and evaluation methods. 3 rd Edition. Pp. 55– 63; 429– 518. Thousand Oaks, CA, USA: Sage Publications, Ltd. González de la Rocha, M. (2008). La vida después de Oportunidades: Impacto del Programa a diez años de su creación. En: Evaluación Externa del Programa Oportunidades. SEDESOL, México DF; 1: 121– 199. Teddlie, C. , & Yu, F. (2007). Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research; 1(1): 77– 100.

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14 -00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc. ; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www. measureevaluation. org