A MultiLevel Framework to Understand Factors Influencing Program

A Multi-Level Framework to Understand Factors Influencing Program Implementation in Schools Celene E. Domitrovich, Ph. D. Penn State Prevention Research Center Implementation Methods Meeting September 20, 2010

Co-Authors & Funding Domitrovich, C. E. , Bradshaw, C. P. , Poduska, J. M. , Hoagwood, K. , Buck; ey, J. A. , Olin, S. , Romanelli, L. H. , Leaf, P. J. , Greenberg, M. T. , & Ialongo, N. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6 -28. NIMH and NIDA (P 30 MH 06624) CDC (1 U 49 CE 000728 and K 01 CE 001333 -01) NIDA (R 01 DA 019984)

Current Challenges l Broad dissemination and “going to scale” with evidence-based intervention l Maintaining effectiveness under “real-world conditions l Understanding the factors that promote or undermine implementation quality throughout the dissemination process

General Definition of Implementation: What an intervention consists of in practice and the degree to which it was conducted as it was originally intended (Durlak, 1995; Yeaton & Sechrest, 1981)

Defining the Model: Intervention Support System Core Elements l Standardization l Delivery l l Core Elements l Standardization l Delivery

Planned vs. Actual Intervention

Planned vs. Actual Support System

Measuring the Quality of the Intervention and Support System l Adherence/Fidelity l Dosage l Quality of Delivery

Factors Outside the Model:

Factors Outside the Implementation Model Individual Level • Professional & Psychological Characteristics • Intervention Perceptions & Attitudes l School Level • Resources, Administrative Leadership and Support • Classroom Climate, School Characteristics l Macro Level • Policies & Financing • Leadership & Human Capital • University/Community Partnerships l

Critical Issue #1: Non-linear process l There are stages to the diffusion of innovations l Implementation quality is critical throughout but very few studies have measured it over time l The factors that support or undermine the process may be more or less relevant at each stage

Critical Issue #2: Factors Interact Factors associated with implementation quality cannot be examined in isolation l They interact both within and across levels l In order to understand how these factors influence one another, we need multiple measures assessed simultaneously and over time. l Large sample sizes are need to address this and many other implementation research challenges l

Critical Issue #3: Measurement l There are different potential sources of implementation data l They vary in how cost effective they are to collect l We know very little about who can assess what , when it should be measured, which measures are the most important, and how often implementation quality should be assessed.

Critical Issue #4: Lack of randomization Most studies of implementation examine associations between factors and implementation or between implementation and target outcomes within the intervention group which limits conclusions that can be drawn. l Interventions often have unique components or processes that are difficult to assess in the control group. Measures that can be used in both conditions provide a way to preserve the RCT design l Very few studies actually test theories of implementation by manipulating implementation itself or factors outside the process in randomized trials l

Critical Issue #5: Multi-component or Integrated Interventions The field is moving towards the use of integrated and adaptive interventions that present unique implementation challenges. l How do we capture quality when multiple implementers are involved in the delivery of an intervention? l How do we understand implementation in the context of interventions where different participants receive different components? l

Future Directions l Theory-driven research – Implementation interventions l Understanding adaptation l Improved Measurement l Statistical strategies to address challenges