Logic Model Workshop Logic Model A Program Performance

Logic Model Workshop Logic Model: A Program Performance Framework Madison Workshop December 5, 2001 Ellen Taylor-Powell, Ph. D. Evaluation Specialist UW- Extension-Cooperative Extension University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop • • A New Era What gets measured gets done If you don’t measure results, you can’t tell success from failure If you can’t see success, you can’t reward it If you can’t reward success, you’re probably rewarding failure University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop • • • If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it If you can demonstrate results, you can win public support Osborne and Gaebler, 1992 in MQ Patton, 1997: 14 University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic Model is… • Picture of a program • Graphic representation of the program “theory” or “action” – what it invests, what it does and what results • Logical chain of if-then relationships; if x occurs, then y will occur • Core of program planning and evaluation University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Ø LOGIC Reasonable To be expected Ø MODEL Represents reality, isn’t reality University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Why Logic Models-Why the Hype? • Shows difference between what we • • do and impact we are having Provides a common vocabulary Focus on quality and continuous improvement University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic Model: Origins • • • Private Sector: Total quality management Public Sector: GPRA, performance budgeting Non-Profit Sector: Outcomes measurement, eg. United Way International Arena: Results Framework of USAID, etc. Evaluators: Evaluability assessment, bennett hierarchy University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic Model S I T U A T I O N INPUTS University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation OUTPUTS OUTCOMES

Logic Model Workshop Everyday Logic Model H E A D A C H E Get pills University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Take pills Feel better

Logic Model Workshop An Extension Example: Business Counseling Extension invests time and resources A variety of educational activities are provided to business owners who participate These owners gain knowledge and change practices resulting in University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Improved business performance

Logic Model Workshop LOGIC MODEL: Program Performance Framework S I T U A T I O N OUTPUTS OUTCOMES-IMPACT Activities Participation Short What we invest What we do Who we reach What the short term medium term ultimate results are impact(s) is Staff Volunteers Time Money Materials Equipment Technology Partners Workshops Meetings Counseling Facilitation Assessments Product dev. Media work Recruitment Training Learning Action Conditions Awareness Knowledge Attitudes Skills Opinions Aspirations Motivations Behavior Practice Decisions Policies Social action Social Economic Civic Environmental INPUTS ASSUMPTIONS 1) 2) University of Wisconsin-Extension 3) Cooperative Extension 4) Program Development & Evaluation Participants Customers Citizens Reactions Medium ENVIRONMENT Influential factors Long-term

Logic Model Workshop Logical Linkages: Series of If-Then Relationships IF THEN Program invests time & money INPUTS IF THEN Resource inventory can be developed OUTPUT University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation IF Families will know what is available THEN IF Families will access services OUTCOMES THEN Families will have needs met

Logic Model Workshop Activity (p 2 -12) Completing the causal linkages University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop LOGIC MODEL: Program Performance INPUTS Program investments What we invest OUTPUTS Activities Participation What we do Who we reach OUTCOMES Short Medium What results SO WHAT? ? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Longterm

Logic Model Workshop Outcomes vs. Activities BE OUTCOME DRIVEN, NOT ACTIVITY DRIVEN University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Activity (p 2 -13) Program Performance Levels University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Situation • Situational analysis – Need – Asset • Problem analysis • Priority setting • Engaging others University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop INPUTS Faculty Staff Money Time Volunteers Partners Equipment Technology University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop OUTPUTS What we do ACTIVITIES Workshops Counseling On-farm research Facilitation Product development Curriculum design Trainings Field days Conferences Media University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Who we reach PARTICIPATION Participants Clients Customers Users

Logic Model Workshop OUTCOMES What results for individuals, businesses, communities. . … SHORT MEDIUM LONG-TERM Learning Action Conditions Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Behavior Practice Decisions Policies Social action Human Economic Civic Environment

Logic Model Workshop INPUTS Staff OUTPUTS Design parent ed curriculum Targeted Money Partners OUTCOMES Parents increase knowledge of child dev parents Provide 6 training session University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation attend Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect

Logic Model Workshop Activity (p 2 -24) Which are the Outcomes-Impacts? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Chain of Outcomes SHORT MEDIUM LONG-TERM Seniors increase knowledge of food contamination risks Practice safe cooling of food; food preparation guidelines Lowered incidence of food borne illness Participants increase Establish financial goals, Reduced debt and knowledge and skills in use spending plan increased savings financial management Community increases understanding of childcare needs Residents and employers Child care needs are met discuss options and implement a plan Empty inner city parking Youth and adults learn Money saved, nutrition lot converted to gardening skills, nutrition, improved, residents enjoy community garden food preparation and mgt. greater sense of community

Logic Model Workshop Focus of Outcomes • Individual Child is prepared to enter school; teen uses savings/spending plan – Child, parent, client, resident • Group – family, team, community group Community group has inclusive membership policy; family increases its savings • Agency, organization Communications are more open; agency adopted smoke-free policy • System Family serving agencies share resources • Community Shared community responsibility has increased; youth are valued as contributing members University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop How far out the outcome chain do we go? • What is logical? • • What is realistic? What is meaningful? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop INPUTS Staff OUTPUTS Develop parent ed curriculum Targeted Money Partners OUTCOMES Parents increase knowledge of child dev parents Deliver 6 interactive sessions Research University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation attend Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect WHICH OUTCOMES? ? ?

Logic Model Workshop Outcome of Interest? • Inherently valued outcome (Mohr, 1995) – higher level outcome is immaterial – we are willing to assume that a higher outcome will also be attained if we achieve the outcome of interest • Participant valued outcome: if participants experience a change or benefit that makes a a real difference to them (United Way, 1999) University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Assumptions • Beliefs about the program – the participants – the way the program will operate – how resources, staff will be engaged – theory of action University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Assumptions, cont. • Faulty assumptions are often the reason for poor results • Check and test assumptions – Identify potential barriers for each then’ sequence University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation ‘if-

Logic Model Workshop Environment–Influential Factors • Extension program does not exist in a vacuum • Context of the program – politics, family circumstances, cultural milieu, demographics, economics, values, biophysical environment, policies, services • What affects the program over which you have little control? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop What does a logic model look like? • • • Graphic display of boxes and arrows Any shape possible Circular, dynamic Relationships, linkages Level of detail • simple • complex • Multiple models University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Compare examples University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic Model: Limitations • Represents reality, not reality üPrograms are not linear • • Focuses on expected outcomes Challenge of causal attribution üMany factors influence outcomes • Doesn’t address: Are we doing the right thing? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Benefits • • Brings detail to broad goals • • Builds understanding and consensus Shows the ‘chain of events that link inputs to results Identifies gaps in logic and uncertain assumptions Signals what to evaluate and when Summarizes complex program to communicate with externals University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Building a Logic Model • • • New program Existing program Team; organization Involvement of others Keep it dynamic University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic Model: WORKSHEET Program Title Situation/Problem INPUTS OUTPUTS Activities ASSUMPTIONS • • University of Wisconsin-Extension Cooperative Extension • Program Development & Evaluation Participation OUTCOMES - IMPACT Short Medium Long-term

Logic Model Workshop Check Your Logic Model • Are the outcomes really outcomes? • Is the longest-term outcome – meaningful? – logical? – realistic? • Are the connections between inputs, outputs, and outcomes clear and reasonable? • Does it represent research and best practice? • Does it represent the program’s purpose; response to the situation? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop PLANNING INPUTS Programmatic investments OUTPUTS Activities Participation Short Medium Long term i EVALUATION University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation OUTCOMES

Providing Leadership for Program Evaluation Where does evaluation fit? INPUTS Staff OUTPUTS Design parent ed curriculum Money Partners Targeted OUTCOMES Parents increase knowledge of child dev parents Provide 6 training sessions attend Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect EVALUATION: What do you want to know? How will you know it? Quality of curriculum # of sessions delivered # parents attending/session which parents % of parents Increase in knowledge/skill - post session survey Actual use - follow-up phone interview Decrease in rates - agency records

Logic Model Workshop Evaluation Plan What do Indicators - Source of Method to you want to how will information collect info know? you know it University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Schedule when/where

Logic Model Workshop Evaluation Questions – What do you want to know? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Indicators - How will you know it? • The evidence or measures that indicates what you wish to know or see: • often multiple indicators are necessary • may be quantitative or qualitative • culturally appropriate University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Logic Model Workshop Logic model with indicators for Outputs and Outcomes Outputs Program implemented Number of workshops held Outcomes Targeted farmers Number and percent of farmers attending University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Farmers learn Number and percent who learned content Farmers practice new techniques Number and percent who practice the recommendatio ns Farm profitability increases Number and percent show farm profits increase; amount of increase

Logic Model Workshop INDICATORS: Examples OUTCOME INDICATORS Increased youth-adult partnerships #, % of Boards with youth participation #, % acres managed according to BMP guidelines Quality of conservation plan implementation #, % with savings goal set #, % with debt reduction goal set #, % using spending plan #, % maintaining emergency fund Reduction in N and P application rates Improved family financial management University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation

Evaluating Your Program INPUTS Staff OUTPUTS OUTCOMES Design parent ed curriculum Targeted Money Partners Parents increase knowledge of child dev parents Provide 6 training sessions attend Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect Indicators Quality of curriculum # sessions fully delivered # and %parents who attended each session #, % parents who increased knowledge #, % parents who learned new ways #, % parents using improved skills (specify skills) Decrease in rates of abuse & neglect among these parents

Logic Model Workshop Source and Method of Data Collection Source of information • • Participants Parents Teachers Key informants University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Method of collecting the information • • Survey Interview Observation End-of-program questionnaire • Focus group • Records

Logic Model Workshop Evaluation Plan cont. How will the data be analyzed and interpreted? University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation How will results be shared? To whom, how, when? Who will do what, when, what resources?

Logic Model Workshop Enfin… • View evaluation as learning - integrate into the way we work • Build evaluation in upfront • Ask ‘tough questions’ • Make measurement meaningful • Be accountable for highest professional standards University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation