Starting Stopping or Sustaining New Empirical Research on

Starting, Stopping, or Sustaining? New Empirical Research on Implementation of SWPBIS Kent Mc. Intosh Rhonda Nese Kathleen Strickland-Cohen Angus Kittelman University of Oregon

The research reported here was supported by the Institute of Education Sciences and Office of Special Education Programs, U. S. Department of Education, through Grants R 324 A 120278, R 324 A 1800027 and H 326 S 180001 to the University of Oregon. The opinions expressed are those of the authors and do not represent views of the Institute, Office, or the U. S. Department of Education.

Overview of Today’s Session 1. Starting (Rhonda Nese) Predicting latency of reaching adequate implementation of tier I SWPBIS 2. Stopping (Kathleen Strickland-Cohen) Variables Contributing to Abandonment & Readoption of SWPBIS 3. Sustaining (Angus Kittelman) Effects of Implementation Strategies on Sustained Implementation of SWPBIS 4. Answer questions at end Handouts: http: //www. pbis. org

Why does this session focus specifically on SWPBIS? Widespread use

U. S. Schools Using PBIS August 2019 27, 294 Schools Using PBIS 30% of all Schools 15, 284, 640 Students

Schools Reporting PBIS Fidelity August 2019

Schools Implementing Tier 1 PBIS with Fidelity August 2019

Why does this session focus specifically on SWPBIS? Widespread use Validated measures of fidelity

SWPBIS Tiered Fidelity Inventory (SWPBIS TFI) Measure of SWPBIS implementation at Tiers 1, 2, and 3 Reliable and valid measure of fidelity (Massar et al. , 2019; Mc. Intosh et al. , 2017; Mercer et al. , 2017) http: //www. pbisapps. org https: //www. pbisapps. org/Applications/Pages/PBISAssessment-Surveys. aspx#tfi

Why does this session focus specifically on SWPBIS? Widespread use Validated measures of fidelity Proven effectiveness

Statistically Significant Outcomes of PBIS Reduced problem behavior (Bradshaw, Mitchell, & Leaf, 2010; Flannery et al. , 2014; Horner et al. , 2005; Lee & Gage, 2020; Metzler et al. , 2001; Nelson, 1996; Nelson et al. , 2002) Increased prosocial behavior (Metzler, Biglan, Rusby, & Sprague, 2001; Nelson et al. , 2002) Improved emotional regulation (Bradshaw et al. , 2012) Improved academic achievement (Horner et al. , 2009; Lassen, Steele, & Sailor, 2006; Nelson et al. , 2002) Improved perceptions of school safety (Horner et al. , 2009) Improved organizational health (Bradshaw et al. , 2008)

Effects of PBIS on Out of School Susps (WI Schools Implementing PBIS; Swain-Bradway et al. , 2019)

STARTING Nese, R. N. T. , Nese, J. F. T. , Mc. Intosh, K. , Mercer, S. H. , & Kittelman, A. (2019). Predicting latency of reaching adequate implementation of tier I school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 21, 106 -116.

Sample 708 schools across 5 States �Strong state networks All of the schools reaching Tier I SWPBIS implementation within 5 years Mostly Title I (61%), Elementary (73%), Suburban (49%), and White (57%)

Research Questions What is the average latency from training to adequate implementation of Tier I SWPBIS across different school characteristics (i. e. , proportion of students eligible for FRL, student enrollment, school level, school locale, Title I status)? To what extent do school characteristics predict latency to reaching adequate Tier I SWPBIS implementation?

Average time to adequate implementation by level? Elementary Middle High 0 0. 5 1 1. 5 2 2. 5 Years to Tier I Implementation 3 3. 5

What about different ELEMENTARY school characteristics? Title I No Title I � 50% FRL <50% FRL Rural Town Suburb City High enroll Low enroll Total Elementary Schools n Mean 369 2. 14 147 1. 82 234 2. 16 282 1. 96 139 2. 30 255 1. 93 52 1. 88 70 2. 11 258 2. 05 516 2. 05 SD 1. 08 0. 93 1. 14 0. 95 1. 11 0. 99 0. 78 1. 19 1. 10 1. 05

What about different MIDDLE school characteristics? Title I No Title I � 50% FRL <50% FRL Rural Town Suburb City High enroll Low enroll Total Middle Schools n Mean 53 2. 75 103 2. 22 37 2. 62 119 2. 34 26 2. 88 78 2. 41 25 2. 04 27 2. 26 78 2. 59 78 2. 22 156 2. 40 SD 1. 39 1. 07 1. 40 1. 14 1. 37 1. 21 1. 06 1. 10 1. 26 1. 14 1. 21

What about different HIGH school characteristics? Title I No Title I � 50% FRL <50% FRL Rural Town Suburb City High enroll Low enroll Total High Schools n Mean 8 2. 38 28 3. 18 3 2. 00 33 3. 09 0— — 15 3. 80 5 2. 80 16 2. 31 18 3. 44 18 2. 56 36 3. 00 SD 1. 06 1. 00 1. 07 1. 01 0. 84 0. 70 1. 15 0. 86 1. 10

Percent How long did it take to reach Tier I implementation? 45 40 35 30 25 20 15 10 5 0 40 30 17 7 1 6 2 3 4 5 Years to Reach Adequate Tier 1 Implementation

What characteristics predicted latency to meet Tier I implementation? Elementary Schools 2. 5 x more likely to reach before middle 8 x more likely to reach before high Non-Title Schools 2 x more likely to reach before Title schools Suburban schools 2 x more likely to reach before city schools

What does this all mean? Adequate implementation took longer than 1 year for most schools This point is critical to district and state teams planning timelines for training Some training schedules = Tier I systems in Year 1, Tier II systems in Year 2, and Tier III in Year 3 This schedule does not allow adequate time to implement the critical features of Tier I SWPBIS As such, moving too quickly from Tier I to Tier II or III trainings may be a substantial barrier to adequate implementation

BUT WHAT ABOUT ABANDONMENT? Nese, R. N. T. , Mc. Intosh, K. , Nese, J. F. T. , Ghemraoui, A. , Bloom, J. , Johnson, N. W. , Phillips, D. , Richter, M. F. , & Hoselton, R. (2016). Predicting abandonment of school-wide behavior support interventions. Behavioral Disorders, 42, 261 -270.

Sample 915 schools trained across 3 States �Strong state networks 1 -7 years of implementation (trained between 2006 -2009) Mostly Title I (95%), Elementary (67%), and Suburban (46%)

Research Questions 1. 2. What proportion of schools abandoned SWPBIS, when, and why? What school characteristics predicted abandonment within the first 5 years of implementation? Level: Elementary, Middle, High Locale: City, Suburb, Town, Rural Title I Year 1 Fidelity Status

What proportion of schools abandoned SWPBIS, when, and why? 40% 35% % of Deactivation 30% 25% 20% 15% 10% 5% 0% 1 2 3 4 Implementation Year Only 7% of the sample abandoned SWPBIS 5

What proportion of schools abandoned SWPBIS, when, and why? Competing Initiatives (4%) Insufficient Data Submission (8%) School Administrator Support (88%) 67% gave a reason for discontinuing implementation

What school characteristics predicted abandonment within the first 5 years of implementation? Level: Elementary, Middle, High Locale: City, Suburb, Town, Rural Title I Year 1 Fidelity Status

Only one significant predictor…

Improved social competence Distal student outcomes SW-PBIS Improved academic achievement Improved student behavior & safety Sustained fidelity of implementation SUSTAINABILITY FACTORS Contextual fit Priority Effectiveness Efficiency Increased Dosage Networking. Principals’ Academies Coalition Building Continuous regeneration Build New Educator Skills Lohrmann & Davis, 2014; Mc. Intosh, Kelm, & Canizal Delabra, 2015

Stopping & Restarting: Variables Contributing to Abandonment and Readoption of SWPBIS Kittelman, Strickland-Cohen, Pinkelman, & Mc. Intosh (2020) Kathleen Strickland-Cohen

Background Transient nature of educational reform efforts can: Present a significant obstacle to lasting improvements in student outcomes Lead to decreases in staff morale and willingness to fully invest in new practices and initiatives (Klingner et al. , 2013) Sustained implementation of large-scale school and district-wide initiatives is challenging (Fixsen et al. , 2017) School- and district-wide reform requires complex systems- level change Regularly scheduled team meetings, data systems to monitor outcomes and fidelity, training and on-going coaching for administrators and staff (Freeman et al. , 2015)

SWPBIS is not only widespread, but also long-lived There are schools and districts that have been implementing SWPBIS for over 15 years Well-established educational initiatives are unique in that schools sometimes abandon these practices, only to be asked or choose to start them again Schools face a unique set of challenges when seeking to readopt practices that have previously been abandoned

Study Purpose Identify schools that had previously stopped SWPBIS only to later readopt in order to determine: What features of SWPBIS were these schools most likely to stop implementing? What were the reasons for abandoning SWPBIS? What were the reasons for readopting SWPBIS?

Participant Sample Participants: 23 school personnel representing 30 schools 10 school-level SWPBIS team members 6 school administrators 4 school personnel who were not SWPBIS team members 3 external district or regional coaches All were in the fourth year of a longitudinal study (n = 330) All indicated that they had abandoned and readopted SWPBIS 53% for less than 1 year 37% for 1 -2 years 7% for 3 -4 years 3% for 5 years or more

Survey Measure & Analysis 9 items: closed- and open-ended questions that addressed: Critical features of SWPBIS that were stopped Amount of time that elapsed between starting and stopping Perceived reasons why SWPBIS practices were stopped The most important factor and why it was so pivotal Perceived reasons why SWPBIS practices were restarted The most important factor and why it was so critical Mixed-methods approach Closed-ended – basic descriptive statistics, mean scores Opened-ended – thematic coding

Critical Features/Practices Stopped Out of 30 Schools Team Meetings (n = 17) Schoolwide Acknowledgment Systems (n = 14) Continuum of Supports for Responding to Problem Behavior (n = 13) Data Collection Systems (n = 12) Collecting Fidelity Data (n = 9) Teaching Schoolwide Expectations (n = 7) All features of SWPBIS (n = 5)

Reasons for Stopping 20 18 17 16 15 15 15 Number of Responses 14 13 12 12 Competing Initiatives Administrator Turnover 12 10 8 6 4 2 0 Staff Buy-in Building Administrator Support Consistency Among Staff Fidelity of Implementation Team Turnover

Reasons for Restarting “The new administrator was a team player and empowered the staff to run the SWPBIS efforts. ” 20 18 16 “The district coach is now meeting with the team on a regular basis. ” Number of Responses 14 12 12 11 “Seeing a school similar in size be successful was an excellent example to staff. ” 10 8 6 5 5 5 4 4 2 0 New Buildinglevel Administrator District Support Time Increased Buy-in Seeing Effectiveness Seeing Other Schools' Results

Discussion of Findings Reports of lack of staff buy-in and administrative support as barriers are consistent with previous studies (Bohanon et al. , 2018; Kincaid et al. , 2007; Nese, 2016; Pinkelman et al. , 2015) School administrator support and district support as facilitators to readoption is consistent with findings from studies examining factors associated with sustained implementation (Forman et al. , 2013; George et al. , 2018; Mc. Intosh et al. , 2013)

Considerations for Restarting: Safeguarding against Re-Stopping Primary reasons given for Readoption were increased school and district administrative support Before restarting, identify and address reasons why SWPBIS was stopped A new champion is great! But… it is important for new administrators to have a clear understanding of: Systems-level barriers Staff-level barriers

Safeguarding against Re-Stopping, cont’d Use past fidelity data, along with new fidelity measures to identify gaps and needs Increase staff buy-in: Survey all staff SAS (PBISApps. org), PBIS-Action and Commitment Tool (Filter et al. , 2016) Anonymous and less formal measures to identify perceived barriers Sharing data (Mc. Intosh et al. , 2015) Do we have a problem that PBIS can help solve? What barriers prevented us from being successful? What will we do to prevent this from happening again? Share data from local demonstration schools, national averages Improve contextual fit

Sustaining: Effects of Implementation Strategies on Sustained Implementation of SWPBIS ANGUS KITTELMAN & KENT MCINTOSH UNIVERSITY OF OREGON 43

Factors Related to Sustainability of SWPBIS District-level (Mc. Intosh et al. , 2013; Mc. Intosh et al. , 2018) § Initiative health § Critical mass § District priority § District capacity building School-level § Grade level § School priority Practice-level § Teams using data § Fidelity in Year 1 44

Factors Affecting Implementation Systems Barriers (Forman et al. , 2009; Pinkelman et al. , 2015) § Turnover § Competitive initiatives § Buy-In Implementation Strategies § Competency drivers § Leadership drivers § Organization drivers - Fixsen et al. 2005; 2008 45

Research Questions: To what extent are Administrator Turnover, Training , Coaching, and Team Use of Data for Decision Making and Tier 1 Implementation Fidelity related over time? METHOD 46

Sample 563 U. S. schools in 11 states across 5 years § Year 1 (2012 -13) to Year 5 (2016 -17) School Locale § § 167 (Cities) 189 (Suburb) 87 (Towns) 120 (Rural) Grade Level § 380 (Elementary) § 108 (Middle) § 58 (High) Stage of Implementation § 123 implementing for 0 -1 years (Initial Implementation) § 273 implementing for 2 -4 years (Institutionalization) § 176 implementing for 5 or more years (Ongoing Evolution) 47

Implementation Strategies & Barriers Training (ADEPT survey) § Categories: None, 1 day or less, 2 to 3 days, 4 to 5 days, over 5 days § Average in Year 1 was 2. 33 to 1. 41 in Year 5 Coaching offered (ADEPT survey) § Categories: none, less than weekly, 1 -2 hours per week, 3 or more hours per week § Average in Year 1 was 1. 46 to 0. 85 in Year 5 Team use of data for decision making (SWIS) § Proportion of months with 10 or more SWIS reports generated § Average in Year 1 was 0. 56 to 0. 49 in Year 5 Administrator turnover (ABISS survey) § Categories: not true, partially true, mostly true, very true § Average score in Year 1 was 1. 37 to 1. 50 in Year 4 48

Tier 1 Implementation Fidelity 49

Analyses Design § Cross-lagged panel modeling Missing data § Maximum likelihood (ML) Commands § Mplus for structural equation modeling § WLSMV for ordered categorical variables § COMPLEX for nested data Model fit § Chi-square goodness-of-fit, RMSEA, WRMR, CFI, & TLI 50

Research Questions: To what extent are Administrator Turnover, Training , Coaching, and Team Use of Data for Decision Making and Tier 1 Implementation Fidelity related over time? RESULTS 51

Administrator Turnover, Training , & Tier 1 Fidelity Findings: § Significant relations among similar variables over time § Training related to fidelity Yrs 3, 4 § Fidelity related to training in Yrs 2 & 3 § Fidelity related to turnover in Yr 2 & Yr 3 52

Administrator Turnover, Coaching, & Tier 1 Fidelity Findings: § Significant relations among similar variables over time § Except coaching Year 1 -2 § Coaching related to fidelity in Yr 3 § Fidelity related to coaching in Yr 2 § Fidelity related to turnover in Yr 2 & 3 53

Administrator Turnover, Team Use of Data for Decision Making , & Tier 1 Fidelity Findings: § Significant relations among similar variables over time § Except Use of Data in Yrs 1 - 2 § Use of Data related to fidelity in Yr 3 & 4 § Fidelity related to Use of Data in Yr 2 § Admin turnover negatively related to fidelity in Yr 3 54

Implications Plan for administrator turnover (Strickland-Cohen et al. , 2014) § Identifying ways to integrate SWPBIS with incoming initiatives Districts as a protective factor (Horner and colleagues, 2018) § Establish proactive district policies § Hire administrators & staff with knowledge of MTSS § Support school teams § Identify efficient ways to measure systems barriers Invest early in implementation strategies (Nese et al. , 2018) § Invest intensely & early to gain momentum § Understand functions of implementation strategies 55

References 1. Fixsen, D. L. , & Blase, K. A. (2008). Drivers framework. Chapel Hill, NC. The National Implementation Research Network, Frank Porter Graham Child Development Institute, University of North Carolina. 2. Fixsen, D. L. , Naoom, S. F. , Blase, K. , Friedman, R. M. , & Wallace, F. (2005). Implementation research: Synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). 3. Forman, S. G. , Olin, S. S. , Hoagwood, K. E. , Crowe, M. , & Saka, N. (2009). Evidence-based interventions in schools: Developers' views of implementation barriers and facilitators. School Mental Health, 1, 26 -36. doi: 10. 1007/s 12310 -0089002 -5 4. Horner, R. H. , & Sugai, G. (2018). Future directions for positive behavior support: A commentary. Journal of Positive Behavior Interventions, 20, 19 -22. doi: 10. 1177/109300717733977 5. Mc. Intosh, K. , Mercer, S. H. , Nese, R. N. T. , Strickland-Cohen, M. K. , Kittelman, A. , Hoselton, R. , & Horner, R. H. (2018). Factors predicting sustained implementation of a universal behavior support framework. Educational Researcher, 47, 307316. doi: 10. 3102/0013189 X 18776975. 6. Mc. Intosh, K. , Mercer, S. H. , Hume, A. E. , Frank, J. L. , Turri, M. G. , & Mathews, S. (2013). Factors related to sustained implementation of schoolwide positive behavior support. Exceptional Children, 79, 293 -311. 7. Nese, R. N. T. , Nese, J. , Mc. Intosh, K. , Mercer, S. H. , & Kittelman, A. (2018). Predicting latency of reaching adequate implementation of school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions. doi: 10. 1177/1098300718783755 8. Pinkelman, S. , Mc. Intosh, K. , Raspica, C. , Berg, T. , & Strickland-Cohen, M. K. (2015). Perceived enablers and barriers related to sustainability of school-wide positive behavioral interventions and supports. Behavioral Disorders, 40, 171 -183 9. Strickland-Cohen, M. K. , Mc. Intosh, K. , & Horner, R. H. (2014). Sustaining effective practices in the face of principal turnover. Teaching Exceptional Children, 46, 18 -24. 56

Contact Information Kent Mc. Intosh Special Education Program University of Oregon kentm@uoregon. edu @_kentmc Cannon Beach, Oregon © Go. Pictures, 2010 Handouts: http: //www. pbis. org