Evaluating Program Sustainability New Visions New Methods Mary

Evaluating Program Sustainability: New Visions, New Methods Mary Ann Scheirer, Ph. D Scheirer Consulting American Evaluation Association, 2014 For copies of slides, contact me at: Maryann@Scheirer. Consulting. com

Is Evaluation Finished? l The Healthy Alternatives Promotion Program for Youth (HAPPY) showed reductions in risky behaviors among youth during its 5 -year funded period in 12 cities. Now what? l In South Africa, a national mass measles vaccination campaign led to a substantial reduction in hospital admissions for measles in the 12 months following the campaign. End campaign?

Outline of Topics for Session I. What is meant by “sustainability”? II. Some data about the extent of sustainment III. Thinking about multiple types of sustainment outcomes IV. What factors are related to sustaining interventions within each organization PSAT? V. Strategies re funding for sustainment

I - What is Program Sustainability? l Definitions differ among researchers - in general, it’s what happens to a program or intervention after external assistance or initial funding ends. l Also called “sustainment” in recent literature – to emphasize looking at it as a long-term outcome l Some writers focus on “sustainability” as a set of capacities or processes, &/or a characteristic of the program.

Multiple Types of “Sustainability” 5 l Economic sustainability - whether an economic development program can maintain itself; often used in international contexts l Environmental sustainability - maintaining the environment in a steady state; not environmentally degrading l Organizational sustainability – does the organization itself continue to exist? l Not the focus of this session!

Context for Program Sustainability: A Program or Intervention Exists l l What is a “program”? - “A set of resources and activities directed toward one or more common goals” Newcomer, et al. , 1994 I assume an organization has implemented a new program or other intervention that it wants to sustain. Programs develop over time, through stages: – Initiation - Why was the program started? Who was/is involved? – Implementation – full or partial? – Sustained use or discontinued? Sustainability is affected by all earlier stages – start planning for it early!

Why Sustainability Matters for Both Evaluation & Practice l Because it’s increasingly important to funders! l Keeping “good will” of community partners, & their willingness to participate in future programs and their evaluation. l Ethical issues arise if partnerships with community organizations are abandoned after research. l Sustainability is itself an important evaluation topic; focus moves from “efficacy” to “effectiveness” in context over a longer term. l “Why bother with what is effective, if it is also fleeting? ” 7

“IN GOD WE TRUST, - ALL OTHERS MUST BRING DATA” Senior NIH Official

II - Data on the Extent of Sustainment – Does it Happen? l I reviewed evaluations of sustainability - in AJE l 14 of 17 studies (82%) reported that 60% or more of their sites sustained their “program”: – l Usually defined as the continued operation of at least some program activities But – Programs that continued were not necessarily at the same level of client benefits

Stirman, et al. Review from 2012 l l l Found 125 studies of sustainability Varied levels found for extent of sustainment; often “partial”. Methods for study weak: “Few studies that included independent observation or validation reported high rates of continuation at the site or setting level. ”

III - What is to Be Sustained? Multiple Sustainability Outcomes l l From Paper by Scheirer & Dearing: 2011 Discuss multiple aspects of evaluation and practice about sustainability: – – – l Definitions of outcomes/dependent variables; Definitions of influences/ independent variables; Research designs & data collection methods. First, focus on specifying outcomes.

6 Types of Potential Sustainability Outcomes a) Benefits/outcomes for clients continued b) Program activities/components maintained c) Community partnerships/coalitions continued d) New policies/procedures maintained e) Dissemination of program components f) Sustained attention to underlying issues

Illustration: l How can these types of sustainability be evaluated in a specific project? l Overview of Project HEAL – “Health through Early Awareness and Learning”

Cheryl L. Holt, University of Maryland, Principal Investigator

Project HEAL l l Uses lay community health advisors (CHAs) in African American churches to lead 3 evidence-based workshops in each church Two study conditions (randomized 14 churches): – Online training for CHAs (n=6 churches) – Traditional in-person training (n=8 churches) CHAs & Pastors recruited workshop participants from church members – 20 - 30 members per church Follow-up sessions 12 & 24 months later

Potential Sustainment Measures for Project HEAL Type of Outcome Measures? Data Source I - Outcomes for clients – a. Have participants been enrolled HEAL church screened? members b. Retention rate for data collection? c. Do participants talk to others? 12 - & 24 month surveys of members II – Continued program activities? - (original components? ) a. Additional Workshops held? b. Other types of workshops? c. Numbers of new participants? Interviews with CHAs - 12 & 24 months III – Maintenance of Partnerships a. TA sought or received from U. of CHA interviews; Maryland? Pastor interviews? b. CHAs meet together – in person CM records or virtual? c. Community Ministry initiates additional activities? IV – Institutionalization of a. Additional resources obtained new church practices, for health activities? procedures, policies b. Integration of health messages into church communications / meetings? CHA interviews

Potential Sustainment Measures for Project HEAL Type of Outcome Measures? Data Source V – Sustained attention to health issues by CHA’s or churches a. Other types of health activities in CHA interviews church increased? b. Continued CHA education on cancer, on other health topics? VI – Diffusion to other sites a. Workshops spread to other CHA interviews churches or sites? Team notes b. New research or partnerships by partners – e. g. Community Ministry? VII – Unplanned, unintended consequences ? ? - To be examined CHA interviews Team notes, anecdotes

Other Issues in Measuring the Extent of Sustainment, as an Outcome 19 l Timing – when to assess whether or not a project is “sustained”? Deteriorate over time? l Is there a threshold for stating that “it” was or was not sustained? l How much “adaptation” or change can occur, and still be the “same” program? l Finding that a program still exists does not mean it was caused by initial funding source.

Your Experience? l Questions about measuring sustainment? l How have you collected data about sustainment?

IV - How to Sustain? What Factors Influence Sustainment? Ø 21 Multiple factors are potential influences on sustainment: § Components of project design and its characteristics § Characteristics of organizational setting § Aspects of community environment Ø No uniform set of factors studied Ø At best, relational analysis done; most from retrospective data collection

Factors influencing sustainability? A. Project design and characteristics l If intervention is modified from its original form; intervention is flexible, adaptable – l Uses no paid staff; uses volunteers; low cost – l Supported by 5 studies Evaluation available & showed effectiveness – 22 Supported by 12 (of 17) studies* – Supported by only 4 studies in my review Reputation for effectiveness likely more influential than results of evaluation

Factors influencing sustainability? B. Factors in Organizational Setting l Good fit between program & host agency’s mission, objectives, and operating routines – l Presence of an internal “champion” – l Supported by 12 (of 17) studies* Supported by 13 (of 17) studies* Organization has strong existing capacity – Supported by 4 studies 23 l Benefits felt by agency &/or staff - 12 studies*

Factors influencing sustainability? C. Community Involvement & Environment l Presence of external funding, type of funding – l Non-monetary support by other community organizations – 24 Supported in 9 studies Found helpful in 12 (of 17) studies*

Logic Model for “Sustainability” Inputs: • Factors Affecting Sustainability Intervention with evidence for efficacy • • Organizational capacity • Prior relationships & the history of innovation Intervention Characteristics Supports • Impl. & Sus. Processes • Environmental support Obtaining Financial Resources Internal üInstitutionalization External üGrants üFee for service In Kind ü Volunteers ü Space, logistics Outcomes: Intervention Sustained üClient Ben efits üIntervention Acti vities üCommunity cap acity üReplication üAttention to issu e Policy and Financial Environment

For details of factors likely to influence sustainability…. Program Sustainability Assessment Tool - PSAT The Program Sustainability Assessment Tool is a copyrighted instrument of Washington University, St Louis, MO. Developed by the Center for Public Health System Science, George Warren Brown School of Social Work, with funding from the Centers for Disease Control and Prevention, Office on Smoking and Health. http: //www. sustaintool. org

© 2012, Washington University in St Louis

V - Funding l“ Whenever anyone says it is not about the money, it is about the money. ” – Attributed to H. L Menken

Financial Strategies: 1. Institutionalization l Transfer from “special” project status to permanent status as an on-going agency activity. l Institutionalization is one type of “sustainability. ” l Assumes that agency has an annual budget from continuing sources, that can fund this activity. l Depends on integration into routines & staffing. l Also called “routinization. ”

Financial Strategies: 2. Health Insurance/ Fee for Service l For example, obtaining coverage under Medicaid l Usually requires extensive negotiations to obtain change in coverage policies l Start early to begin decision-making processes l Fees from clients – What would they be willing/able to pay? l May be applicable for client services, training, other ?

Strategies: 3. External Grants or In-Kind l Funding sources: – – – – Gov’t grants Local foundations Corporations Budget shared by partners Fund raising events Bequests Individual donations l In-kind resources – – – – Personnel time Recruit partners w/ needed skills Meeting/office space Supplies Lists of volunteers Networking contacts Recruit retired professionals w/ needed skills Sustained programs often have multiple sources of

A Typical Plan for Sustainability?

Some Conclusions l Sustainability is possible, and may be frequent! l Outcome variables need to be defined – what outcome is to be sustained? l Sustainment of different types of program components may have different influences or predictors. l Contingency models needed, to relate program type to specific predictor variables and to environments. l Multiple types of data collection are needed, across time.

And… Do Remember:

Additional Resources l l International context - tools & examples: – CEDARS – Center for Design & Research on Sustainability - www. cedarscenter. com – Implementation Research Toolkit – www. who. int/tdr/publications/topics/ir-toolkit/en/ Program Sustainability Assessment Tool l www. sustaintool. org

That’s All, Folks! Questions?

References l Scheirer, M. A. Linking Sustainability Research to Intervention Types. American Journal of Public Health, 103 (2013) #4, pp. e 73 -e 80. l Scheirer, M. A. & J. W. Dearing. An Agenda for Research on the Sustainability of Public Health Programs. American Journal of Public Health. 101 (2011) 2059 - 2067. l Scheirer, M. A. , Hartling, G. and Hagerman, D. B. , Defining Sustainability Outcomes of Health Programs: Illustrations from an On-line Survey. Evaluation & Program Planning 31 (2008) 335 -346. l Scheirer, M. A. , Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability. American Journal of Evaluation, 26 (2005) 320 -347. l Schell, S. F. Luke, D. A. et al. Public health program capacity for sustainability: Anew framework. Implementation Science, 8: 15 (2013). See also: http: //www. sustaintool. org