EvidenceInformed Program Improvement Using principles of effectiveness to

Evidence-Informed Program Improvement: Using principles of effectiveness to enhance the quality and impact of youth and family programs STEPHEN SMALL Professor of Human Development & Family Studies, University of Wisconsin-Madison and Extension Human Development & Family Relations Specialist University of Wisconsin-Extension Iowa State Webinar November 17, 2009

Agenda I. II. III. What are evidence-based programs, practices, and organizations? What are some common principles of effective evidence-based youth and family programs? How can Evidence-Based Program Improvement (EIPI) be used to improve existing programs?

Evidence-based Programs and Practices Years of research have demonstrated that specific approaches, practices and strategies can reduce problem behaviors and enhance positive developmental outcomes The most effective programs and practices are termed “evidence-based” – these are the “gold standard”

Why the growing interest in evidencebased programs and practices? • Accountability: Increasingly required by funders • • • within political climate Efficiency: Reduces development costs associated with creating a new program Availability of good science: Body of scientific evidence has reached a critical mass Cost-effectiveness: Increased likelihood that limited resources will be used wisely Marketability: Documented evidence can help sell and justify program Ethics: Responsibility to use what is known to be effective

The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels and core elements Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

Evidence-Based Programs A new class of programs that: • Are based on a sound scientific evidence and theory • Have been carefully implemented, evaluated and replicated using rigorous scientific methods • Have evaluation findings that have been subjected to critical review • Often “endorsed” as evidence-based by a federal agencies or respected research organizations (e. g. , Campbell Collaboration, SAMHSA, CDC)

Terminology Evidence-based Research-based Evidence-based = Research-based + Rigorously tested

Examples of Evidence-based Programs Family/Parent Education Programs Nurse Family Partnership: 15 -year follow-up showed ↓ arrests (54%) among adolescents; $3. 59 return on the dollar Strengthening Families Program for Parents and Youth 10 -14: ↓ aggression, alcohol, tobacco, marijuana, meth; $7. 82 return

Examples of Evidence-based Programs School-based programs Life Skills Training: 6 -year follow-up showed ↓ alcohol, tobacco, marijuana, narcotics, hallucinogens; $25. 61 return on the dollar Olweus Bullying Prevention Program: ↓ in bullying and victimization, vandalism, fighting, truancy; ↑ in school attachment

Examples of Evidence-based Programs Mentoring Big Brothers Big Sisters: ↓ in violence, alcohol and drug use; ↑ in school performance; $1. 01 return

Aggression and Hostility Index The effect of the Strengthening Families 10 -14 Program on teen aggressive and hostile behaviors SOURCE: Spoth, R. , Redmond, C. , & Shin, C. (2000) Reducing adolescents' aggressive and hostile behaviors: Randomized trial effects of a brief family intervention four years past baseline. Archives of Pediatrics and Adolescent Medicine 154, 1248 -1257

What are some limitations of evidence-based programs? Not always easy to find Can be costly to implement in both time & $ May not address particular issues or audiences Tend to be problem focused May downplay local knowledge and community ownership Overlooks value of promising grassroots programs Often difficult to transport to new settings Other?

The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels and core elements Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

Evidence-Based Kernels • Kernels = units of behavioral influence and practice that have been shown through experimental evaluation to produce reliable effects on behavior • Kernels = Active ingredients of effective programs

The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

Definitions of Evidence-based practices “The integration of the best available research with clinical expertise in the context of patient characteristics, culture and preferences. ” (American Psychological Association) “The integration of best-researched evidence and clinical expertise with patient values. ” (Institute of Medicine of the National Academies)

Evidence-Based Practice Practitioner Expertise & Experience Evidence. Based Practices Researc h Evidenc e

The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

EVIDENCE-INFORMED PROGRAM IMPROVEMENT: Using principles of effectiveness to enhance the quality and impact of family-based prevention programs February 2009, Family Relations, 58, 1 -13. http: //whatworks. uwex. edu

Principles of evidence-based youth and family programs 1. Design & Content 2. Relevance 3. Delivery/Implementation 4. Assessment & Quality Assurance

Program design and content Effective programs… Target relevant assets and/or risk and protective factors Are theory-driven Are of sufficient dosage and intensity Are comprehensive Use active learning approaches & appropriate delivery formats

Theory-driven Based on strong, scientific theory Target relevant risk, protective factors and assets to accomplish objectives Logical, well-developed program theory A well thought out recipe is essential

Theory driven - Example Nurse-Family Partnership (early childhood/parent ed) - grounded in theories of human ecology (Bronfenbrenner, 1979, 1992), self-efficacy (Bandura, 1977, 1982), and human attachment (Bowlby, 1969)

Theory-driven Based on strong, scientific theory Target relevant risk, protective factors and assets to accomplish objectives Logical, well-developed program theory A well thought out recipe is essential

Have sufficient dosage and intensity Enduring change takes effort

Thinking about sufficient dosage and intensity… For a mentoring program to be effective, the relationship on average has to exist for at least… A. 2 weeks B. 1 month C. 6 months D. 9 months E. 1 year

Comprehensive Simple solutions rarely work

Use active learning techniques and appropriate delivery format

Program relevance Effective programs… Are developmentally appropriate Are socio-culturally relevant Are appropriately timed Are sensitive to local context Are responsive to individuals One size does not fit all

Program delivery Effective programs… • • Foster good relationships Are delivered by well-trained and committed staff Address environmental barriers to program delivery and behavior change -Provide a safe setting for the intervention Are embedded in a supportive organization infrastructure

Well trained and committed staff - Examples Big Brothers, Big Sisters (mentoring program) – Orientation, volunteer screening, youth assessments, matches, supervision Promoting Alternative Thinking Strategies (PATHS, classroom program) - Teachers receive training in a two- to three-day workshop and in biweekly meetings with the curriculum consultant.

Program assessment and quality assurance Effective programs… Are well-documented Focus on evaluation and refinement Provide opportunities for ongoing feedback

Make sure the program is well-documented It’s important to understand the program in order to implement it well Maintaining program fidelity requires that people know what the program is about Program replication and evaluation is difficult if the program isn’t well documented

Focus on evaluation & refinement Ø See evaluation as your friend Ø Evaluation takes many forms Ø Evaluation isn’t always easy Ø Consider EIPI as a first step

Evidence Informed Program Improvement http: //whatworks. uwex. edu/attachment/whatworks_manual. pdf

SUMMARY Principles of Effective Youth and Family Prevention Programs Program design and content Program relevance Program delivery Program assessment and quality assurance

The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

Evidence-based organization • Promotes an organizational culture that values science and evidence-based approaches • Creates an infrastructure that supports effective practices and programs: • Uses existing evidence-based programs when available and • • • appropriate Employs a strong theoretical/science approach to new program design Provides regular opportunities to reflect on programs and effective practices Uses scientific tools to assess needs/assets, understand local context and adapt programs Provides ongoing technical support for selecting, implementing and evaluating effective programs and practices Builds organizational and community capacity to identify relevant

Questions/Discussion