How the Funder Can Structure a Useful Evaluation

How the Funder Can Structure a Useful Evaluation Laura C. Leviton, Ph. D. Special Adviser for Evaluation The Robert Wood Johnson Foundation

Overview Goal: How to produce a more useful, high quality product Learn from my mistakes and successes Discuss your situation – any applicability? Part I: RWJF does useful evaluations regularly Part 2: An ongoing process of improvement in evaluation

One Funder Only-- Me A very particular context Lots of evaluation, taken seriously What might apply to • philanthropy? • government?

The Robert Wood Johnson Foundation Builds the field of health • Health insurance for all Americans • Improve systems of care • Tobacco control • Public health infrastructure • Childhood obesity prevention

The Robert Wood Johnson Foundation A learning organization • 35 years of – evaluation – policy analysis – related research • 30 evaluations ongoing in any given year • We use the results. – Not always, but often. Marketing of Unhealthy Foods Pervasive in U. S. Public Schools Survey Finds Financing End-of-Life Care: Challenges for An Aging Population Thinking Strategically About Collaboration: Tobacco and Alcohol Control Policy

The Purpose of Evaluation in Foundations Some Stated purposes: • “Accountability” (Pew) • “Program Improvement / Grantee Empowerment” (WKKF) • “Learning” (RWJF)

Learning or Accountability? We choose learning. Accountability is important BUT Accountability induces fear. • Fear impairs discussion • Fear impairs learning • We separate accountability from evaluation

Learning or Individual Grantee Empowerment? Strong internal evaluation findings are good. Grantee capacity is good. We do all this. However we have to ask: What is the quality of information for the FIELD? Will the aggregate results of individual evaluations lead to strategic change? We choose deliberate cross site evaluation to assist strategy • A priori questions where possible. • NOT some post hoc cluster thing.

Useful Evaluations, Produced Regularly Background - Foundation Strategy Case studies: – Summative review for Foundation board, staff, the field: • End of Life Care – Outcome evaluation to persuade policy makers: • Cash and Counseling – Selection of promising practices for the field: • Childhood Obesity Prevention

What is Foundation Strategy? A plan to achieve change, using tactics such as: • Increase attention to a problem • Advocate for policy • Stimulate innovation • Support professional training • Build capacity of organizations, programs, communities, coalitions, etc.

Case Study: Improving Health Care at the End of Life

Patients Do NOT Get Good Care in the US

Purpose of the Summative Review Document: • Foundation strategy as it emerged • Improvements in health care practice See how strategy was actually created Describe lessons for foundations Source: Patrizi Associates, 2008

Method of the Study Used Donald Schon’s “Reflective Practitioner” approach: • “what was your rationale at the time? ” Reviewed 327 grants: 1991 to 2007 Interviewed 70 leaders in the field Reviewed all “seminal” documents Source: Patrizi Associates, 2008

Approaches to Improve EOL Care 1. Improve knowledge and capacity of health care providers to care for the dying ($26 million) 2. Improve EOL care in health care institutions and advance EOL public policy ($64 million) 3. Engage the public and professionals to improve EOL care ($79 million) We will discuss the first 2 areas– the third was VERY mixed! Source: Patrizi Associates, 2008

Improving Professional Knowledge • Educate physicians in end-of-life care $6. 5 million, 1997 -200 • Educate nursing students in end-of-Life care $6. 4 million, 2000 -2004 • Insert EOL content on licensure exams $8. 7 million, 1998 -2004 • Develop formal standards for palliative care in VA $1 million, 1998 -2003 Source: Patrizi Associates, 2008

Improving Professional Knowledge Results: • Failed to train all physicians in EOL • Required EOL content on exams • Reformed medical and nursing education Source: Patrizi Associates, 2008

Creating Institutional Support • Find innovative models $15 million, 1996 -2005 • Develop palliative care in hospitals $29 million, 1999 -2008 • Reform state pain policy $4. 3 million, 2000 -2007 • Improve institutional quality $7. 3 million, 1997 -2006 Source: Patrizi Associates, 2008

Creating Institutional Support Results: Found innovative models but failed to exploit lessons Spread hospital based palliative care Adopted policy changes Source: Patrizi Associates, 2008

State of the EOL Field Today End-of-life care has improved… • Almost 3, 000 physicians and over 15, 000 nurses are certified in palliative care • 70 percent of hospitals with 250+ beds have palliative care units (30 percent of all hospitals) • Palliative care became a medical subspecialty in 2006 • States’ pain policies have improved We hope to learn. . . • How better to integrate the component parts of strategy. Source: Patrizi Associates, 2008

Case: Selecting Promising Practices Foundations stimulate innovation and variation: “Let 1000 flowers bloom” Why not have evaluation select the effective variants?

Case: Selecting Promising Practices Evolutionary epistemology (Campbell) • Variation: funded innovations • Selection: evaluation • Dissemination: spread of selected variants

Evaluations are Usually Handed to Us “Here, evaluate this. ”

Some of the Programs Just Aren’t Very Good “Here, evaluate this. ”

Identify a High Volume of Programs

Then Screen Them for Prior Probability of Impact

Then Test Them Childhood Obesity: 458 nominations 128 presented to expert panel to assess • plausibility of impact • likely reach to cover target population • likely feasibility, generalizability, etc. 53 deemed worth of evaluability assessment • 48 assessed Yield from the process: • 33 ready for evaluation • 20 both ready AND likely to have population impact • 6 priorities, 5 evaluations under way or approved.

Forthcoming monograph: The Systematic Screening & Assessment Method: Finding Innovations Worth Evaluating. New Directions in Evaluation, No. 125 January 2010 Leviton, Kettel Khan & Dawkins, Editors SSA Model

In Brief: Strategic Outcome Evaluation Case study: Cash and Counseling • The culmination of years of innovation • Involvement of people with disabilities as stakeholders • A strategic reason to examine outcomes • A randomized experiment with a strategic purpose

People with Disabilities Want Independent Choices BUT: Medicaid pays for personal care • bathing, feeding, etc. Medicaid assigns a stranger to do this • from a social service agency If the choice were left to the person with disability • would the quality of life increase? • would there be fraud and abuse? • would costs remain the same?

Results of Cash and Counseling Better quality of life, cost neutral Convinced Medicaid and 30 states to date.

How Can We Get There? Predictable process + The right fit between program and evaluation + Funder as coach.

Improved Process A process to improve this production, relying on Greater predictability Stages of program development and correct use of evaluation for those stages Identifying evaluator strengths and weaknesses Getting the right fit Leveraging evaluators’ intrinsic motivation

Predictable Process 1. Anticipate evaluation needs 2. Have potential evaluators on tap. 3. Plan evaluation early 4. Adjust expectations 5. Clarify and re-clarify purpose to all concerned.

Anticipate! Anticipate evaluation needs • Evaluation staff participate in program creation • Rolling list of evaluations upcoming • Consult internal and external stakeholders regularly Evaluators on tap • Evaluator database • Thorough vetting: report review • Identify strengths and weaknesses for different assignments in d A F J W See R s of e g a p t s a l ram g o r p A E A

Plan Evaluation Early. Ideally: Work together from the very start of the program planning Choose evaluation team at same time as Program Director Involve Program Director in evaluation discussions and planning logistics Choose advisory members with evaluation expertise Independence of evaluation does not equal firewall.

Clarify and Reclarify Purpose To program applicants and funded programs: • Describe general requirements in some detail • Make evaluation FAQ available • Describe binding requirement to participate in specific evaluation activities. Provide detail where possible: • Timeline • Requirements • Budget implications

Clarify and Re-clarify : • Purpose of evaluation • What to expect, and when • How grantees can benefit • What will be required • Roles and responsibilities

Manage Expectations: Flexibility • Programs evolve, so must evaluation • Logistics -- negotiated where need be Constructive information Binding requirement for specific activities • Right there in the letter of award Mutual respect • “We must be responsive to our grantees and to the field. ” • Guiding Principles for Evaluators

The Right Fit: Program and Evaluation Maturity of the Field Idea Generation Refine the Ideas Evauation Focus Formative - basic Formative - advanced Ideas are Well. Understood Test Impact Spread & Sustain Models Replication/Adaptation/ Quality Control Cross-Program Strategy Assessment

The Right Fit, Evaluator Attitude It Is Not the Critic Who Counts “It behooves every man to remember that the work of the critic is of altogether secondary importance, and that, in the end, progress is accomplished by the man who does things. ” • Theodore Roosevelt

The Right Fit: Evaluator Strengths Scope of work, interests Strengths Content expertise Temperament for the particular work. In all cases, high quality.

The Right Fit: Firmness + Flexibility “How do you handle ambiguity? ” Sample responses: • “I sit down with them to go over the logic model. ” • “Negotiation and finding common interest. ” • “Chuckle” • “Badly” • No response.

Funder as Coach Program Grantees Evaluation Grantee or Contractor RWJF and External Advisors 44

The Funder as Coach Power Point