DESIGNING AND CONDUCTING CASE STUDIES FOR DEVELOPMENT EVALUATIONS

DESIGNING AND CONDUCTING CASE STUDIES FOR DEVELOPMENT EVALUATIONS A Preconference Workshop for the IDEAS Global Assembly in Johannesburg March 17, 2009 Linda Morra Imas Lmorra@worldbank. org © IPDET 2009 lindag 1000@aol. com

Workshop Objectives • Understand defining features of case studies • Know when to use case studies and when each type of case study is appropriate • Contrast case study methods • Know how to increase case study reliability and validity, and determine sample size • Apply some case study methods © IPDET 2009 2

BOOKS TO READ 1. Robert E. Stake (1995). The Art of Case Study Research, SAGE Pub. , Thousand Oaks: CA. 2. Robert K. Yin (2005). Case Study Research: Design & Methods, Third Ed. Applied Social Research Methods Series, Volume 5. SAGE Pub. , Thousand Oaks: CA. 3. Robert E. Stake (2006). Multiple Case Study Analysis, Guilford Press, New York: New York. www. amazon. com © IPDET 2009 3

Experience with Case Studies • Who has conducted case studies? • How did you know it was a case study? © IPDET 2009 4

Definition A case study is a method of learning about a complex instance, based on a comprehensive understanding of that instance obtained through extensive description and analysis of that instance taken as a whole and in its context. © IPDET 2009 5

Discussion • A case study is a method of learning about a complex instance…. . – Simple instance vs. complex instance • Based on a comprehensive understanding – Fly through qualify? • Obtained through extensive description and analysis – Qualitative, not quantitative? • Taken as a whole and in its context – ? ? ? ? © IPDET 2009 6

Case Studies versus Other Approaches • Histories: no access or control– rely on documents and artifacts (the past). • Experiments: done when behavior can be manipulated – a social experiment. • Surveys: large # of cases; thin descriptions; empirical generalizations © IPDET 2009 7

DEFINING THE CASE STUDY Case studies emphasize thick descriptions – Not just descriptions of people, places, and events – But also the interpretations of those people most knowledgeable – Evaluator applies principle of relativity in identifying the best explanations © IPDET 2009 8

CASE STUDY PURPOSE “The real business of case study is particularization, not generalization. We take a particular case and come to know it well…There is emphasis on uniqueness…we emphasize placing an observer in the field to observe the workings of the case, one who records objectively what is happening but simultaneously examines its meaning and redirects observation to refine or substantiate those meanings. ” Robert E. Stake. 1995, pg. 8 -9 © IPDET 2009 9

THE APPROACH • • • Holistic view of complex instances Observation Progressive focusing Searching for patterns Developing assertions © IPDET 2009 10

When to Use A Case Study • Preferred strategy for “how” or “why” questions. • Can use for “what” questions, but …. – “Who” or “where” questions or “how many” or “how much” or “extent” questions favor survey strategies or analysis of archival records. © IPDET 2009 11

Would You Use Case Study ? • If you want to know “how frequently” project sites are obtaining monthly target outputs, would you likely use a case study? • If you want to know more about cases that illustrate a certain pattern (e. g. sites not meeting targets), is case study appropriate? © IPDET 2009 12

Would You Use Case Study ? • If you want to know what a best or worst case looks like and what makes it different, would case study be appropriate? • Would you use case study to answer “How many clients did the program serve? ” Or “What kinds of benefits were received most often? ” • If you want to know what a typical case looks like, would you use case 13 study? © IPDET 2009

Use Case Study • If you want to know “why” or “how” the program worked (or did not). • Key: exploratory or explanatory purpose; not a frequency or extent purpose. © IPDET 2009 14

CASE STUDIES Myth or Fact © IPDET 2009 15

#1 Myth or Fact? Case studies take a long time. © IPDET 2009 16

# 2 Myth or Fact? Case studies only use qualitative methods. © IPDET 2009 17

# 3 Myth or Fact? Case studies provide little basis for generalization. © IPDET 2009 18

# 4 Myth or Fact? You can use case studies to draw causal inferences. © IPDET 2009 19

THREE CASE STUDY CATEGORIES © IPDET 2009 20

Three Case Study Categories • Descriptive: describe an intervention and the context in which it occurred. • Explanatory: explain causal links in interventions; link program implementation with program effects. • Combined : Brings together findings from several case studies to answer an evaluation question. Overlap in real life! © IPDET 2009 21

3 Types of Descriptive Case Studies 1) Illustrative – Add realism and in-depth examples to other information about a program, project, or policy. – Describe what is happening and why, to show what a situation is like. – Often used to help interpret survey data. – Generally selected as typical or representative of important variations. – Number generally kept small. © IPDET 2009 22

Types of Descriptive Case Studies 2) Exploratory To explore those situations where little is known about the intervention or its potential outcomes. © IPDET 2009 23

Types of Descriptive Case Studies 3) Critical Instance To examine a single instance of unique interest or serve as a critical test of an assertion about a program, project, problem, or strategy. © IPDET 2009 24

2 Types of Explanatory Case Studies 1) Program Implementation: The case study investigates operations, often at several sites, and often normatively. 2) Program Effects: The case study examines causality and usually involves multisite and multimethod assessments. © IPDET 2009 25

1 Type of Combined Case Study 1. Cumulative: Brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause and effect. © IPDET 2009 THINK SYNTHESIS! © IPDET 2009 26

EXERCISE 1. Is case study appropriate? 2. If yes, what type? © IPDET 2009 27

DESIGNING THE CASE STUDY EVALUATION © IPDET 2009 28

Doing Case Studies Prepare Define Design Analyze Conclude Collect © IPDET 2009 29

Identify Evaluation issue Conduct 1 st Case study Write case report Develop theory Select cases 2 nd Conduct case study Write case report Draw cross-case conclusions Modify theory Develop policy implications Develop design Write data collection protocol Conduct remaining case studies Write case reports Case Study Method: Adapted from Yin, 2003. Pg. 50. Write crosscase report 30

DEFINING THE EVALUATION ISSUE • Why is this particular program being evaluated at this particular time? • Who is the main client and what are their major concerns and questions? • How will they use the evaluation results? • Who are the major stakeholders and what are their concerns and questions? © IPDET 2009 31

DEVELOP THEORY OF CHANGE • Conduct a literature review • Talk with experts • Engage stakeholders in the process of articulating the program theory • Develop conceptual model of how program/project/intervention is supposed to work = TOC • Highlight assumptions and relationships between the key elements © IPDET 2009 32

Develop Design • Design: the overall strategy for systematically gathering and analyzing data to address the main evaluation purpose or issue. • Begins with questions identified through stakeholder analysis and TOC/logic model. • Is case study indicated? – General approach? – Depends also on the types of questions posed. © IPDET 2009 33

Questions and Case Study Design • How does the new village well typically affect the lives of the women in the village? • What types of outcomes both intended and nonintended are associated with the Rural Roads Expansion Program? Illustrative Case Study Exploratory Case Study © IPDET 2009 34

Questions and Case Study Design • How critically important is the support group to the success of the women’s Microenterprise Program? Critical Instance Case Study © IPDET 2009 • How are women using the food and nutritional supplements provided by the provincial health clinics? Program Implementation Case Study 35

Questions and Case Study Design • Did educational services provided by the agriculture extension center result in planting of the new grain seed and increased yields? Program Effects Case Study • Does the body of evaluation literature indicate that the building of roads increases trade and benefits villages economically? Cumulative Case Study 36 © IPDET 2009

Using Multiple Strategies • More than one strategy can be used in a study. • Which comes first, the survey or the case study? [Sam D. Sieber. “The Integration of Fieldwork and Survey Methods”. The American Journal of Sociology, Vol. 78, No. 6, (May, 1973), pp. 1335 -1359. http: //www. jstor. org/stable/2776390] 37 © IPDET 2009

DEFINING THE SAMPLE © IPDET 2009 38

MULTIPLE CASES NEEDED WHEN – Bracketing. What is happening at the extremes? What explains differences? – Best cases/Worst cases. Why is (isn’t) it working? – Cluster. How do different types of instances compare with each other? – Representative. How do different types of instances chosen to represent important variations compare and why? © IPDET 2009 39

Multiple Case Studies • Same study contains more than a single case. – For example, Study of school innovations. Independent innovations occur at different sites. – Every case should have a specific purpose. • Often needed for explanatory case studies. Replication - same results predicted for each case or contrasting results for predictable reasons. 40 © IPDET 2009

SINGLE CASE APPROPRIATE – Exploratory. In this particular circumstance, what is happening and why? Could be an extreme or unique case. – Typical. In a typical site, what is happening and why? – Critical instance. Does this single instance support theory? © IPDET 2009 41

Embedded Case Studies • Within a single case (single public program), the analysis includes outcomes from individual projects within the program. • Embedded units selected through sampling or cluster techniques. (Project characteristics) • More frequent versus holistic approach. © IPDET 2009 42

Other Sampling Strategies • Probability Sample: What is happening as a whole and why? – Not appropriate for case study selection, but can use sampling for embedded units. • Convenience Sample: In this site, selected for data collection convenience, what is happening and why? – Not a good sampling strategy © IPDET 2009 43

Defining the Sampling Unit • Family/kinship group • Small group – E. g. What is a community? • Geographic area – E. g. How define boundaries of a neighborhood? • Time period © IPDET 2009 44

GENERALIZING FROM THE CASE STUDY SAMPLE © IPDET 2009 45

GENERALIZING FROM THE SINGLE CASE STUDY? • Naturalistic generalization. Not scientific, but intuitive, based on personal or vicarious direct experience (Stake) • Transferability and fittingness. The “fit” or similarity between two cases determines the transferability of conclusions from one to the other (Lincoln & Guba) © IPDET 2009 46

GENERALIZING FROM MULTIPLE CASE STUDIES Replication • A finding emerging repeatedly across study of numerous sites yields a good working hypothesis • The more different the sites, the stronger the working hypothesis © IPDET 2009 47

Analytic Generalization Robert Yin: Ruling out rival hypotheses • If two or more cases are shown to support the same theory, replication may be claimed. • The empirical results are stronger if two or more cases support the same theory but do not support an equally plausible, rival theory. © IPDET 2009 48

Illustration • Proposition: Increase in number of days of school attendance per year for girls in rural areas will occur when both extra income is provided to their families and the girls’ personal safety is assured. • Select 3 -4 sites in which both interventions are present. • Select 3 -4 sites where only extra income is provided to the family, with the prediction being little increase in girls’ school attendance. • Might select 3 -4 sites where only physical safety is assured, with the prediction being little increase in girls’ school attendance. © IPDET 2009 49

DATA COLLECTION METHODS © IPDET 2009 50

Data Collection Methods • See handout on data collection options • Three key approaches – Observation – Interview – Document analysis © IPDET 2009 51

Data Collection Methods OBSERVATION – Participant – Non-Participant – Obtrusive – Unobtrusive Group Review of Observation DCI © IPDET 2009 52

Form & Exercise Data Collection Methods: Paired Observation Parts I & II © IPDET 2009 53

Data Collection Methods • INTERVIEW – Individual – Small Group – Large Group © IPDET 2009 54

Data Collection Methods A GOOD INTERVIEW IS A GOOD CONVERSATION! AND, WHAT MAKES FOR A GOOD CONVERSATION? © IPDET 2009 55

Data Collection Methods • DOCUMENT ANALYSIS – Contemporary records – Historical records – Personal information – Public information © IPDET 2009 56

TRIANGULATION: Checking for Consistency Increases Credibility • Methods triangulation – Most common – Using different data collection methods • Data triangulation – Using different data sources • Investigator triangulation – Using several evaluators © IPDET 2009 57

Convergence of Multiple Sources of Evidence Documents Archival Records Interviews FACT Observations Focus Groups Physical Artifacts Source: Adapted from Yin, 2003. Pg. 100 58

EXERCISE Case study Critiques © IPDET 2009 59

PREPARING FOR THE CASE STUDY © IPDET 2009 60

Increasing Reliability • Reliability: if the study was repeated the same results • Increase: use case study protocols • TRAIN & TRAIN! • Help develop case study data bases © IPDET 2009 61

Selecting Data Collectors • Field Investigators and Skills Needed – Asking questions – Listening – Flexibility – Understanding of issues being studied – Objectivity © IPDET 2009 62

Training – – – – – READ DISCUSS PRACTICE! why the study is being done theoretical background basic concepts terminology study issues evidence being sought case study methods and tactics data collection and analysis procedures reporting requirements © IPDET 2009 63

Case Study Protocol Contains the procedures and general rules to be followed. Increases the reliability of the evaluation. Anticipates problems that might arise and identifies solutions or strategies. © IPDET 2009 65

Case Study Protocol “Must Haves” 2 Page Overview 1. Study background 2. Study purpose and objectives 3. Study scope and methodology 4. Case study issues, relevant readings © IPDET 2009 1 Page Letter of Introduction 1. Sponsoring org. , official, title and contact detail 2. Study title 3. Data collector name, org. , and assignment 66

DATA ANALYSIS © IPDET 2009 70

Data Analysis • • • Coding schemes Cross validation Disconfirming evidence Themes Patterns Group Review of Observation Analysis © IPDET 2009 71

DATA ANALYSIS • Holistic fallacy • Using both qualitative and quantitative evidence • Computer software • Within and between sites © IPDET 2009 72

Exercise Data Analysis: Paired Observation III © IPDET 2009 73

Content Analysis • Ethnograph 5. 03 www. qualisresearch. com – Searches, marks with code words, runs analysis • atlas. ti 5. 2 www. atlasti. com – Searches recorded multimedia plus print • Nvivo 8 (formerly NUD*IST) www. qsrinternational. com/products_nvivo. aspx – Also multimedia, PDF files, and most languages • MAXqda 2007 (formerly win. MAX) www. maxqda. com 74 © IPDET 2009

Content Analysis Continued • Lewins, A. & Silver, C. (2007) Using Software in Qualitative Research: A Step-By-Step Guide, SAGE Pub. , London • Comparison of software © IPDET 2009 75

PRESENTING FINDINGS © IPDET 2009 76

STRATEGIES FOR PRESENTING FINDINGS • • Natural history Critical incident Thematic Within and between © IPDET 2009 77

PRESENTATION FROM WHOSE VANTAGE POINT DOES ONE REPORT? © IPDET 2009 78

PRESENTATION • The power of pictures • The power of video, but is it worth it? © IPDET 2009 79

SUMMARY The Good Case Study… • Design follows operational specification of the issue (e. g. , a logic model) • Discovery and flexibility are accompanied by re-design • Rival explanations are addressed • Evidence (data) and interpretation are clearly distinguished • Evidence is from multiple sources © IPDET 2009 80

GET REAL LINDA! OK, SO NOW I KNOW ALL ABOUT DOING GOOD CASE STUDIES. BUT REALITY IS THAT I HAVE ABOUT ONE WEEK MAX TO SPEND AT EACH OF 4 SITES. WHAT CAN I DO? © IPDET 2009 81

Use your new knowledge to strengthen your methodology but note limitations © IPDET 2009 82

How Did We Do? 1. Understand defining features of case studies 2. Know when to use case studies and when each type of case study is appropriate 3. Contrast case study methods 4. Know how to increase case study reliability and validity, and determine sample size 5. Apply some case study methods © IPDET 2009 83