Building an Organizational Evaluation Infrastructure Culture to Improve

Building an Organizational Evaluation Infrastructure / Culture to Improve the Management and Outcomes of Health Programs Judith Hager, MA, MPH Molly Bradshaw, MPH Geraldine Oliva, MD, MPH Family Health Outcomes Project University of California, San Francisco October 24, 2001

An Evaluation Infrastructure An institutional philosophy / ethic to provide quality, relevant, cost effective services Shared systematic approach / framework Staff with an understanding of and basic evaluation skills Access to evaluation resources (tools, Information, data, etc. )

Why an Evaluation Infrastructure Ø A Formal System of Accountability Ø Influence External Decision Making Ø Assist Program Management Decisions How to best use resources, improve client and staff satisfaction Ø Promote team-building

Evaluation Is: z. The systematic investigation of the quality and effectiveness of organized efforts or activities z. The systematic assessment of the merit, worth, or significance of an object (CDC)

A Program is Any organized set of activities intended to achieve an outcome (supported by resources) Can be an Initiative, Departmental Program or a Project

Three approaches to building an evaluation infrastructure y. State MCH Sponsored Evaluation Course y. Non-profit Community Collaboration Framework y. County Health Department capacitybuilding

MCH “Evaluation Course” for County MCH Teams Ø Two workshop days / county teams Ø Choose and develop a “real” program evaluate Ø Technical assistance by phone or onsite Ø Assignments and feedback to

Solano Coalition for Better Health - (5013 C) Ø Comprised of hospitals, health plans, health depart. , CBOs, comm. Clinics Ø Contract for development / facilitation / data collection & analysis Ø Three goals / comm. health indicators Ø Agree to share info

Mendocino County Health Department Ø Mandate for all Division/program managers Ø Philosophy of program improvement / resource allocation Ø 2 day workshop Ø Develop evaluations /group critique Ø Manuals / train the trainers

Elements of An Evaluation Infrastructure Ø Ø Ø Ø Mandate Purpose Model/Approach Training Expertise Data Capability Gestalt / Integrated

Evaluation Planning Framework (CDC) Ensure Use Share Lessons Learned Engage Stakeholders Justify Conclusions Gather and Analyze Evidence Describe the Program Focus the Evaluation Design

Creating and Using a Logic Model Purposes: Understand the program, how it works, desired results and Test logic

More about why……. • • • Examine / improve broad, fuzzy objectives Show the ‘chain of events’ linking inputs to results Clarify difference between activities and outcomes Identify gaps in logic / assumptions Help determine what to evaluate / key ? ’s Build understanding & consensus about the program

Building a Logic Model Evaluation logic models: Ødepict how a program works to achieve its intended outcomes Ømay be a flow chart, table, diagram, etc Øhave common elements

Elements of an Evaluation Logic Model z z Inputs (resources) Activities (interventions) Outputs Outcomes (short, intermediate and longterm) * Some models also include other elements such as problem statement, assumptions, environment and program target (e. g. , 13 to 17 year olds / age, sex, location)

Today, we will use an adaptation of the UWEX* logic Model A graphic representation that shows logical relationships between inputs, outputs and outcomes relative to a situation Elements: Ø Ø Ø Problem statement/ situation Inputs Outcomes Assumptions Environment * Ellen Taylor-Powell, University of Wisconsin - Extension

Program Logic Model: Preparation Ø Ø Ø Problem Statement Program Description Goals and Objectives Stakeholders / Use of Evaluation Program concepts / action theory Literature Review

UWEX LOGIC MODEL S I T U A T I O N INPUTS ASSUMPTIONS (Theories that guide your program) 1. 2. 3. OUTPUTS OUTCOMES ENVIRONMENT (external factors)

What it tells us INPUTS OUTPUTS Programmatic investments Activities Resources allocated Effort / What program does Participation Who program targets OUTCOMES Short Inter- Long mediate term With what results

PLANNING INPUTS Programmatic investments OUTPUTS Activities Participation OUTCOMES Short Medium Long term EVALUATION

INPUTS STAFF (special requirements) MONEY LOCATION VOLUNTEERS PARTNERS EQUIPMENT TECHNOLOGY

OUTPUTS What program does ACTIVITIES Treatment Classes Counseling Case management Curriculum design Trainings Conferences Who it reaches PARTICIPATION Participants Providers Users

OUTCOMES What results for individuals, agencies, communities. . … SHORT Intermediate Learning Action Access Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation Behavior Practice Decisions Policies Systems change LONG-TERM Conditions Mortality Morbidity Quality of Life Environmental

Logical Linkages: Example Series of If-Then Relationships IF THEN Program invests time & money INPUTS IF THEN Design parenting curriculum OUTPUTS IF Parents increase knowledge THEN IF Parenting improved OUTCOMES THEN Decrease rates of child abuse

Program Example Problem: Child Abuse INPUTS Staff OUTPUTS Design parent ed. curriculum Targeted Money Partners OUTCOMES Parents increase knowledge of child develop. parents Provide 6 training session attend Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect

Where Does Evaluation Fit? INPUTS Staff OUTPUTS Design parent ed. curriculum parents Money Partners Targeted Provide 6 training sessions attend OUTCOMES Parents increase Parents knowledge/ child develop. use improved parenting Parents skills learn new ways to discipline Reduced rates of child abuse & neglect EVALUATION: What do you want to know? What data do you need? Quality of curriculum # parents attending session Increase in knowledge/skill Behavior change Decrease in rates # of sessions delivered demographics of parents Pre-post session survey follow-up interview/ob agency records

Elements of An Evaluation Infrastructure Ø Ø Ø Ø Mandate Purpose Model/Approach Training Expertise Data Capability Gestalt / Integrated

Limitations of an evaluation infrastructure: z. Requires staff commitment / time z. Difficult to evaluate own work z. May not support rigorous evaluation z Not the answer to management problems z Still need access to expertise

Recommendations – Leadership build trust – Develop a common language – Introduce evaluation model – Provide training

Recommendations – Build a team approach – Provide evaluation tools – Allocate time / long term – Provide guidelines about what is important to evaluate

Recommendations z Understand Evaluation is Complex x. Accountability for the denominator population x. Understand stages of evaluation z Know when consultants work best

Program Planning and Evaluation