ScalingUp Innovative Interventions Fidelity of Implementation and Sustainability

Scaling-Up Innovative Interventions Fidelity of Implementation and Sustainability Andrew Buck, M. A. Envision. IT is funded by the U. S. Department of Education, Office of Special Education Programs, Grant Award H 327 S 120022. The opinions expressed herein do not necessarily reflect the policy or position of the U. S. Department of Education, and no official endorsement should be inferred.

Scaling-Up Innovative Interventions Objectives • How our project has defined, measured, and enforced fidelity • How fidelity has challenged implementation and informed sustainability

Stepping-Up Technology Implementation Project The Envision. IT Curriculum

Stepping-Up Technology Implementation Evidenced-Based 1 Lombardi, A. R. , Izzo, M. V. , Gelbar, N. , Murray, A. , Buck, A. , Johnson, V. , Hsiao, J. , Wei, Y. , & Kowitt, J. (2017). Leveraging information technology literacy to enhance college and career readiness for secondary students with disabilities. Journal of Vocational Rehabilitation, 46(3), 389 -397. Available at https: //content. iospress. com/articles/journal-of-vocational-rehabilitation/jvr 875. 2 Lombardi, A. R. , Izzo, M. V. , Rifenbark, G. , Murray, A. , Buck, A. , Monahan, J. , & Gelbar, N. (2017). The impact of an online transition curriculum on secondary student reading: A multilevel examination. Career Development and Transition for Exceptional Individuals, 40(1), 15 -24, Available at: http: //journals. sagepub. com/doi/full/10. 1177/2165143416681287. 3 Izzo, M. V. , Yurick, A. , Nagaraja, H. N. , & Novak, J. A. (2010). Effects of a 21 st-Century curriculum on students’ information technology and transition skills. Career Development for Exceptional Individuals, 33 (2), 95 -105. Available at http: //journals. sagepub. com/doi/pdf/10. 1177/0885728810369348.

Stepping-Up Technology Implementation Standards-Aligned Envision. IT Google Drive Library: Standards Alignment

Why Envision. IT? • Free, flexible, and optimized for blended & digital learning courses • Full 12 -unit curriculum and course models for students with and without disabilities in 7 th – 12 th grades Supports federal law, state policy, and individualized learning plans • • Teacher-led, student-developed, Transition Portfolios (e. g. , transition assessment results, career research, postsecondary goals, resume and cover letter template, digital presentation) • Created with accessibility & Universal Design for Learning in mind

Fidelity of Implementation Defined, Measured, and Enforced

Defining Fidelity • • • Fidelity of Implementation The American College Dictionary • Strict observance of promises, duties, etc. … • Loyalty Merriam-Webster • The quality or state of being faithful • Accuracy in details: EXACTNESS Fidelity Criteria: Development, Measurement, and Validation (Mowbray et al. , 2003) • The extent to which delivery of an intervention adheres to the protocol or program model • Significant for evaluation, treatment effectiveness, and service administration

Defining Fidelity • • • Fidelity of Implementation Content Validity • Is the tool measuring your research questions? • Are the measurements capturing the knowledge you seek? Measurement Reliability • Does the tool measure results with consistency for all users? • Can different users perceive the scales and terms similarly? Variability • Is the tool sensitive enough to measure change over time? • How much variance does the tool anticipate and collect?

Measuring Fidelity • Fidelity of Implementation Teacher Fidelity Checklist • Pedagogical best practices • Ohio Department of Education: Teacher Performance Evaluation Rubric • Connecticut Department of Education: Common Core of Teaching Rubric for Effective Teaching • Curriculum specific practices • Qualitative classroom observations • Formal scoring protocol • Self-assessment for reflection, inter-rater reliability, and coaching

Measuring Fidelity • Teacher Lesson Logs (i. e. , dosage) • • Checklist of core components • Transition Portfolio activities • Supporting content and activities Qualitative teacher notes • • Fidelity of Implementation Additional content and activities (e. g. , 21 st Century Wednesdays) Classroom Coding (i. e. , setting) • Dedicated • Integrated • Resource Room

If educators are following best practices for effective teaching identified by state departments of education, will they be implementing curricular interventions with high levels of fidelity?

Should a fidelity measurement be more specific to a particular intervention or more generalized to meet the needs of the field?

Enforcing Fidelity • • • Fidelity of Implementation Classroom Observations • Schedule initial visit to gather baseline data • Once per month or until consecutive passing scores (i. e. , >75%) Coaching • Build rapport; observations are not judgements • Written, verbal, and visual feedback • Highlight strengths and identify areas for improvement Technical Assistance • On-going communication • On-site visits and scheduled webinars

Does a variety of items and measurements (e. g. , 2 -pt scale, 4 -pt scale, qualitative responses) triangulate data or create complexity across observers?

How do you ensure that the protocol is being followed precisely by different observers with varying perspectives across sites and states?

Implementation Science Installation and Initial Implementation

Fidelity of Implementation Installation Stage Figure adapted from Duda, M. , Sims, B. , Fixsen, D. & Blase, K. (2012). Making it happen with active implementation frameworks: Improvement cycles. TACC Implementation Webinar. Retrieved from http: //sisep. fpg. unc. edu/sites/sisep. fpg. unc. edu/files/resources/SISEP-Webinar. Series-Session 3 -Cycles-12 -2012. pdf

Fidelity of Implementation Installation Stage Year 1: 2012 -13 Year 2: 2013 -14 Year 3: 2014 -15 Year 4: 2015 -16 Year 5: 2016 -17 Training Coaching Initial training agenda; focus on fidelity Structured yet customized agendas and schedules Establish utility, competency, and fidelity Improved training; communicate updates Recruit additional users Support mentors to coach and develop resources Integrate training model into organizations (e. g. , local, state, private Schedule visits and maintain communication Evaluating Structured yet customized coaching and support Gather and assess outcomes Collect and evaluate data Foster rapport, confidence, and support Improve training and coaching based on data Improved coaching; strengthen relationships Collect and evaluate data and feedback Recruit mentors to coach and develop resources Improve training and coaching based on data Recruit leaders to drive formal installation and sustainability efforts Integrate coaching model into organizations (e. g. , local, state, private Finalize protocol to collect and evaluate data Leverage data to produce articles, disseminate information, and support

Fidelity of Implementation Installation Stage Initial Training Fidelity Visits Technical Assistance • On-site computer lab or classroom setting • Schedule times with teacher to observe class • Maintain regular communication with sites • Structured, yet customized, agenda • Complete fidelity checklist and assess coaching needs • Address specific issues quickly on case-by-case basis • Interactive to promote practice and group work • Ask teacher to complete a fidelity checklist as self-assessment • Post general responses and updates to group online • Professional development resources and supports • Respond to questions, gather feedback, provide support and coaching • Encourage group to share resources, ideas, questions and feedback online • End goal is ability to use intervention effectively • End goal is improving fidelity and rapport • End goal is enhancing support and capacity

Fidelity of Implementation Installation Stage The keys for success in this stage include: • Establishing a multi-level implementation team • Building rapport with stakeholders across implementation years • Addressing feedback with diligence to resolve issues and consider recommendations • Adapting to challenges and barriers during installation and implementation

Initial Implementation Stage Fidelity of Implementation Figure adapted from Fixsen, D. , Blase, K. , Naoom, S. & Duda, M. (2015). Implementation drivers: Assessing best practices. National Implementation Research Network. Retrieved from http: //implementation. fpg. unc. edu/sites/implementation. fpg. unc. edu/files/NIRNImplementation. Drivers. Assessing. Best. Practices. pdf

Initial Implementation Activities with a Rural School Implementation Driver Activity Enact communication protocols Competency Execute coaching plan Measure and report outcomes Leadership Review recommendations for process improvement Maintain regular communication Assess fidelity and review results with teachers Provide immediate feedback Gauge student outcomes through pre- and post quizzes Gauge teacher performance with assessment Update training, coaching, and fidelity checklist Differentiate curriculum for students with and without disabilities and across grades Implement curriculum in an inclusive classroom, such as a freshman seminar elective Organization Measure and report fidelity Schedule site visits to observe and assess fidelity Collect pre- and post-test data and present findings to key personnel Table adapted from Blase, K. , van Dyke, M. & Fixsen, D. (2013). Stages of implementation analysis: Where are we? National Implementation Research Network. Retrieved from http: //implementation. fpg. unc. edu/sites/implementation. fpg. unc. edu/files/NIRN-Stages. Of. Implementation. Analysis. Where. Are. We. pdf

Initial Implementation Stage Fidelity of Implementation The keys for success in this stage include: • Developing practitioner competencies required to use the intervention with fidelity • Aligning the intervention with organizational roles, functions, and goals • Providing support and leadership to address issues and overcome constraints to implementation • Assessing data to improve iterative implementation cycles, including training and coaching

Full Implementation Stage Fidelity of Implementation Concerns-Based Adoption Model – Case Study • The school district needed a high school level curriculum that would address the transition planning needs of students with and without an IEP. • The transition planning course needed to be the length of one semester and was worth one-half (0. 5) elective credit. • One special education teacher and one general education teacher would be selected as instructors (separate classrooms). • The school district used the Schoology learning management system (LMS) and needed a transition curriculum that would work with the LMS. • The curriculum needed to address career exploration, training/education planning, career planning, and support technology literacy to address new state requirements.

In your projects, who have been the catalysts to support and secure an implementation of your intervention?

Sustainability Recruiting, Supporting and Building Capacity

Recruiting Partnerships • • Sustainability Anticipate Attrition • Recruit twice as many as needed to begin; supports RCT designs • Sites may be used as control groups or waitlisted to scale-up Marketing Materials • Make a case for your intervention amongst competing priorities • Research results, alignment to federal laws and standards • How to preview the intervention before committing to implementation

Recruiting Partnerships • • • Sustainability Request for Proposals • Budget for research teams in other states (e. g. , State Do. E and IHE) • Build a model for implementation and sustainability in diverse settings Memorandums of Understanding • Cooperative agreements that establish expectations & accountability • Multi-tier team building: administrative and educator signatures Multi-Dimensional Recruitment • Educator Administrator; School District • Administrator Educators; District Schools

Supporting Partnerships • Professional Development • Initial training, observation, fidelity, coaching, and technical assistance • Static resources and dynamic peer-to-peer PD network, online platform • Scheduled webinars; platform specific and general technical assistance • Sustainability Survey Measures • Training evaluation • Teacher technology use (pre and post) • Teacher satisfaction • Services needed • Sustainability

Supporting Partnerships • Sustainability Survey Measures, continued • • • Training evaluation • Identify helpful methods and resources • Addresses GPRA measures Teacher technology use (pre and post) • Identify users requiring the highest level of support • Addresses CBAM Change Facilitator Style of administrators Teacher satisfaction • Identify strengths and weaknesses of intervention • Addresses GPRA measures

Supporting Partnerships • Sustainability Survey Measures, continued • • Services needed • Identify need for support, training, and technical assistance • Tool targeted new users Sustainability • Identify current users and intervention champions • Identify need for support , training, and technical assistance • Tool targeted users across years, including those who left project • Tool will also be deployed to previous contacts whose implementation status is unknown • Also works as recruitment tool for all previous contacts

Building Capacity • Sustainability Synergistic Opportunities • Ohio • • • State Support Teams and Educational Service Centers • Career Advising Policy and Ohio Means Jobs website • Career Counseling Pilot and Innovative Strategies Grants • Building PD modules in educators’ LMS to earn CEUs Connecticut • State Educational Resource Center and Transition PD Group • CORE Transition Skills and Student Success Plans New York • Career Development and Occupational Studies Credential

Building Capacity • Sustainability Dissemination Strategies • Accessible & adaptable resources for delivery across multiple systems • Digital resource repositories and cloud-based storage systems • Leverage existing networks across levels (LEAs, SEAs, IHEs, national associations – AUCD, NTACT, UCEDDs) • Professional development workshops with regional partners • Conference presentations

Challenges • Sustainability Key Personnel Turnover • District and school administrators (e. g. , director of pupil services, principals) and educators • Transition Planning vs. Career Exploration • Transition planning is perceived as a special education service • • Career exploration is perceived as a general education elective • • Guaranteed for students with IEPS or 504 plans Not required for graduation Competing Resources and Priorities • Career Cruising, Project Discovery, Overcoming Obstacles • High stakes testing and graduation requirements

Challenges • • • Sustainability Course Offerings and Personnel • Official courses/electives dedicated to career development • Semester vs. year-long vs. advisory period course models • Integration of intervention into school system/structure/culture • Staff availability Blended and Digital Learning Pedagogy • Access to reliable educational technology resources • Educator and student familiarity with educational technology Achieving Maximum Potential • Translating students’ Transition Portfolios into IEPs/SSPs • Collaboration between general and special education

Amongst many competing priorities for schools, how do you make a case that your intervention should be adopted and sustained?

Fidelity and Sustainability Balancing Project Priorities and Purpose

Balancing Project Priorities Fidelity Sustainability Generalizability Replication Protocol Uniformity Accessibility Cost per Pupil Relevance

Balancing Project Priorities Sustainability Fidelity Sustainability Customizability Generalizability Replication Protocol Accessibility Cost per Pupil Relevance

How does fidelity of implementation affect an innovative intervention that requires a flexible and customizable approach to secure sustainability across diverse sites?

“How do we determine which critical ingredients are essential to the model and the expected outcomes relative to those that may be adapted, omitted, or added at any given site? ” (Mowbray et al. , 2003, p. 335)

References and Resources • Active Implementation Hub: http: //implementation. fpg. unc. edu/ • Blase, K. , van Dyke, M. & Fixsen, D. (2013). Stages of implementation analysis: Where are we? National Implementation Research Network. Retrieved from http: //implementation. fpg. unc. edu/sites/implementation. fpg. unc. edu/files/NIRNStages. Of. Implementation. Analysis. Where. Are. We. pdf • Cohort 8 Technology Implementation Projects: http: //mdcc. sri. com/cohort 8. html • Connecticut Department of Education (2014). Common Core of Teaching (CCT) Rubric for Effective Teaching: http: //www. connecticutseed. org/wpcontent/uploads/2014/05/CCT_Rubric_for_Effective_Teaching-May_2014. pdf • Duda, M. , Sims, B. , Fixsen, D. & Blase, K. (2012). Making it happen with active implementation frameworks: Improvement cycles. TACC Implementation Webinar. Retrieved from http: //sisep. fpg. unc. edu/sites/sisep. fpg. unc. edu/files/resources/SISEP-Webinar. Series-Session 3 Cycles-12 -2012. pdf The Envision. IT (EIT) curriculum and related products were produced under the U. S. Department of Education (USDOE), Office of Special Education Programs Grants H 327 A 020037, H 327 A 050103, and H 327 S 120022. The views expressed herein do not necessarily represent the policy or position of the USDOE, and no official endorsement by the Department should be inferred. For use or adaptation, please review the EIT Copyright/Reference Disclaimer (http: //go. osu. edu/eitdisclaimer).

References and Resources • Fixsen, D. , Blase, K. , Naoom, S. & Duda, M. (2015). Implementation drivers: Assessing best practices. National Implementation Research Network. Retrieved from http: //implementation. fpg. unc. edu/sites/implementation. fpg. unc. edu/files/NIRNImplementation. Drivers. Assessing. Best. Practices. pdf • Mowbray et al. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation. 24(3), 315 -340. Retrieved from http: //journals. sagepub. com/doi/abs/10. 1177/109821400302400303 • Ohio Department of Education (2015). Teacher Performance Evaluation Rubric: http: //education. ohio. gov/Topics/Teaching/Educator-Evaluation-System/Ohio-s-Teacher-Evaluation. System/Teacher-Performance-Ratings • State Implementation and Scaling-Up of Evidence-Based Practices Center (SISEP): http: //sisep. fpg. unc. edu/ The Envision. IT (EIT) curriculum and related products were produced under the U. S. Department of Education (USDOE), Office of Special Education Programs Grants H 327 A 020037, H 327 A 050103, and H 327 S 120022. The views expressed herein do not necessarily represent the policy or position of the USDOE, and no official endorsement by the Department should be inferred. For use or adaptation, please review the EIT Copyright/Reference Disclaimer (http: //go. osu. edu/eitdisclaimer).

Thank You! Dr. Margo Izzo, Principal Investigator (margo. izzo@osumc. edu) Alexa Murray, Program Manager (alexa. murray@osumc. edu) Andrew Buck, Learning and Development Consultant (andrew. buck@osumc. edu)