DEFINE what is to be evaluated Simon Hearn
DEFINE what is to be evaluated Simon Hearn, ODI, London Part 2 of 8 AEA Coffee Break Webinars 2013
The Rainbow Framework
Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr
Three evaluation tasks in DEFINING Identify potential unintended results Develop program theory or logic model Develop initial description
1. Develop initial description
Peak Experience Thumbnail description Aims & Objectives Vision & Mission
2. Develop program theory or logic model
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
INPUTS • Apples • People at risk of poor health IMMEDIATE RESULTS SHORT-TERM RESULTS LONGER-TERM RESULTS • Apples eaten • Improved nutritional status • Improved health
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
RESULTS AREA INDICATOR MEANS OF VERIFICATION ASSUMPTIONS/ RISKS GOAL: Decreased sick days School and work records Good status largely due to nutrition PURPOSE: Adequate levels of Vitamin C Sample blood tests Apples retain Vitamin C OUTPUT: Apples replaced by apple cores Visual inspection Apples eaten on location Required numbers delivered Delivery receipts Required numbers bought Purchase receipts Visual inspection Improved health Improved nutritional status Apples eaten ACTIVITIES: Deliver apples ACTIVITIES: Buy apples
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Outcomes chain showing possible alternative causal paths Improved health Improved vitamin C levels Improved quercetin levels Decreased Body Mass Index Apples eaten (whole or juice) Red apple skin eaten Apples eaten instead of junk food snacks Apples delivered to schools
Options for representing logic models Pipeline / results chain Logical framework Outcomes hierarchy / theory of change Realist Matrix
Context Mechanism Outcome Healthy people at high risk of gastric cancer Quercetin from red apple skin (or red onion) protect cells from free radicals Reduced risk of cancer Smokers or people with regular exposure to radiation Chemicals in cigarette smoke or radiation damages DNA in healthy cells Continued increased risk of cancer Individuals with vitamin deficiency through malnutrition Sufficient vitamin C absorbed when juice from apples (or oranges) consumed Stronger immune system Individuals with vitamin deficiency through excessive alcohol Alcohol destroys vitamin C No change in vitamin C level Individuals with obesity due to excessive snacking on high fat, high calorie foods Decreased calorie intake due to substitution of apples (or carrot sticks) Reduced obesity and related conditions Decrease sugar intake through substitution of apples No change in obesity but lower risk of diabetes Individuals with obesity due to genetic condition
Context Mechanism Outcome Healthy people at high risk of gastric cancer Quercetin from red apple skin (or red onion) protect cells from free radicals Reduced risk of cancer Smokers or people with regular exposure to radiation Chemicals in cigarette smoke or radiation damages DNA in healthy cells Continued increased risk of cancer Individuals with vitamin deficiency through malnutrition Sufficient vitamin C absorbed when juice from apples (or oranges) consumed Stronger immune system Individuals with vitamin deficiency through excessive alcohol Alcohol destroys vitamin C No change in vitamin C level Individuals with obesity due to excessive snacking on high fat, high calorie foods Decreased calorie intake due to substitution of apples (or carrot sticks) Reduced obesity and related conditions Decrease sugar intake through substitution of apples No change in obesity but lower risk of diabetes Individuals with obesity due to genetic condition
http: //betterevaluation. org/resource/example/rubrics-oakden
3. Identify potential unintended or negative results Source: www. smarttoolkit. net
Negative program theory Key informant interviews Six Hats Thinking Risk assessment
Defining – Framing iteration Initial description Agree scope of evaluation DEFINE Discuss evaluation purpose FRAME Develop logic model Refine description DEFINE Define evaluation questions FRAME
http: //betterevaluation. org/plan/define facebook. com/betterevaluation @bettereval Simon Hearn Research Fellow, ODI s. hearn@odi. org. uk
- Slides: 23