Building Reporting District ProgramLevel Evaluation of PBIS LEAD

Building & Reporting District & Program-Level Evaluation of PBIS LEAD PRESENTER: JAKE OLSEN, PHD EXEMPLARS: AIMEE CLARK & LYNN NORD KEY WORDS: APPLIED EVALUATION, ASSESSMENT, TRAINING 3 National PBIS Leadership Forum – October 4 -5, 2018 – Chicago, IL

Session Objectives 1. Gain a clear understanding of the process and practices involved in PBIS program evaluation 2. Learn about evaluation tools and how to use these tools in your local context 3. Gain insight into the benefits and challenges of district level PBIS program evaluation from the perspective of practitioners

Maximizing Your Session Participation When Working In Your Team Consider 4 questions: – – Where are we in our implementation? What do I hope to learn? What did I learn? What will I do with what I learned?

Where are you in the implementation process? Adapted from Fixsen & Blase, 2005 Exploration & Adoption • We think we know what we need so we are planning to move forward (evidencebased) Installation • Let’s make sure we’re ready to implement (capacity infrastructure) Initial Implementation • Let’s give it a try & evaluate (demonstration) Full Implementation • That worked, let’s do it for real and implement all tiers across all schools (investment) • Let’s make it our way of doing business & sustain implementation (institutionalized use)

Leadership Team Action Planning Worksheets: Steps Self-Assessment: Accomplishments & Priorities Leadership Team Action Planning Worksheet Session Assignments & Notes: High Priorities Team Member Note-Taking Worksheet Action Planning: Enhancements & Improvements Leadership Team Action Planning Worksheet

Please Provide Feedback Your feedback is important to us! Please take a few moments at the end of the session to complete an evaluation form for this session. Forms are available: • In our mobile application by clicking the link in the session description. • Online underneath the posted presentations at www. pbis. org/presentations/chicago_forum_17 • Via paper form from your session facilitator

Purpose of PBIS Program Evaluation §Determine impact of systems and practices §Document implementation process 1. Are goals being met? 2. Are appropriate activities being used? 3. Is process being implemented as intended? 4. Is process successful? (Algozzine et al. , 2010)

Evaluation Process and Practices Shift from Large scale, multi-measure, broad Time and personnel intensive Evaluation as observation Shift to Targeted evaluation Efficient evaluation tools Evaluation as active part of implementation process (Algozzine et al. , 2010)

Core Features of Program Evaluation (Algozzine, 2015)

Evaluation Tools Tiered Fidelity Inventory §Measures extent to which school personnel are implementing core features of Tiers I, II, and III of PBIS §Intended for evaluation every 3 rd or 4 th team meeting §Based on existing evaluation tools (SET, Bo. Q, TIC, SAS, BAT, MATT) §Efficient, valid, reliable §Also has walkthrough component, observation form, action planning form

Reporting Evaluation Results Considerations §Breadth and depth determined by: §Needs of audience §Length of PBIS implementation §Complexity of results data/information (Algozzine et al. , 2010)

Reporting Evaluation Results Considerations §Audience §Most appropriate timing §Effective communication/writing styles §Appearance of reports §Accuracy, balance, fairness §Nature of information (positive, negative, neutral) (Algozzine et al. , 2010)

Reporting Evaluation Results Least Interactive Most Interactive § Newsletters, § Verbal Presentation § Working Sessions Bulletins, Briefs, § Video Presentation § Video Conferencing Brochures § Pictures § Personal Discussion § Website § Executive Summary § Interim Reports § Final Reports (Fitzpatrick et al. , 2011)

Reporting Evaluation Results Written Report Recommended Content 1. Executive summary 2. Purpose, timeframe, summary 3. Vision, mission, foundations of SWPBIS program 4. Current implementation efforts 5. Extent of implementation 6. Outcomes 7. Capacity 8. Implications and next steps 9. Supporting materials (Algozzine et al. , 2010)

Reporting Evaluation Results Oral Reporting Recommendations 1. Determine the story you want to tell 2. Decide who should tell the story 3. Consider varying the medium used in order to engage audience 4. Use visuals to accompany oral reporting 5. Involve the audience 6. Develop and adhere to an agenda

Exemplar HEMET UNIFIED SCHOOL DISTRICT HEMET, CA

Plan IDENTIFY PURPOSE, TEAM, AND TIMELINE

Purpose §Decrease office discipline referrals §Reduce suspensions and expulsions §Increase school attendance §Increase positive perceptions among district students and staff regarding school climate §Implement all three tiers of the PBIS framework district wide corresponds with HUSD’s goals from the

Team § § § Student Support Services Director Grant Project Director Two District PBIS Coaches District Intervention Counselor District MFT District MH Clinical Coordinator

Timeline 2010 -14 Tentative start with practices but not systems 2012 AB 1729 requiring interventions prior to suspension for specific student discipline issues 2014 -19 2014 -15 2015 School Climate Transformation Initiative Award Contracted with California Technical Assistance Center (Cal. TAC) for implementation of the National Model Dr. Rob Horner makes formal presentation to District Leadership

Timeline 20152016 • Cohort 1: Tier 1 Implementation • Cohort 2: Exploration & Installation 20162017 • Cohort 1: Tier 2 Implementation and Tier 1 Sustainment with Boosters • Cohort 2: Tier 1 Implementation 20172018 • Cohort 1: Tier 3 Implementation and Tiers 1 & 2 Sustainment with Boosters • Cohort 2: Tier 2 Implementation and Tier 1 Sustainment with Boosters • Restorative Practices 20182019 • Cohort 1: All Tiers Sustainment with Boosters • Cohort 2: Tier 3 Implementation and Tiers 1 & 2 Sustainment with Boosters • Restorative Practices • Mindfulness

Define IDENTIFY QUESTIONS TO BE ANSWERED

Evaluation Questions §How can HUSD reduce discipline referrals, suspensions, and expulsions? §How can the district improve systems to promote environments that are conducive to learning? §How can HUSD improve school climate, safety, and connectedness to improve academic achievement?

Do SELECT MEASURES AND GATHER DATA TO ANSWER QUESTIONS

Tiered Fidelity Inventory (TFI)

Data Gathering Monday TFI Walk. Through Schedule February 5 -16, 2018 Tuesday Wednesday Thursday Friday 2/5: West Valley HS @8: 30 Cawston ES @10: 30 2/6: Bautista Creek @11: 00 Little Lake @12: 30 Acacia MS @12: 45 2/7: NA 2/8: NA 2/9: Ramona ES @8: 30 Dartmouth MS @8: 30 Whittier ES @10: 00 Rancho Viejo @10: 30 2/12: Hamilton K-8 @8: 30 Helen Hunt J. @9: 30 Hamilton HS @10: 00 2/13: Fruitvale ES @8: 30 Alessandro HS @9: 00 Tahquitz HS @10: 30 2/14: Harmony ES @8: 30 Idyllwild K-8 @8: 30 Mc. Sweeney @10: 00 Cottonwood @10: 30 2/15: Hemet HS @9: 00 Valle Vista ES @ 12: 00 Diamond Valley @1: 30 2/16: Hemet Elem. @8: 00 Jacob Wiens ES @9: 30 Winchester ES @11: 00

Data Gathering §Self-Assessment Survey (SAS) §Self-assessment tool to determine to what extent PBIS practices and systems are in place within a school §Four systems: School-wide, Non-Classroom settings, Classroom, Individual Student §Goal: 80% In Place School-wide

Document ANALYZE DATA AND ANSWER QUESTIONS

Self Assessment Survey (SAS)

Answering Evaluation Questions Through the process of implementing PBIS district-wide: § Worked with the student information system Aeries to create a PBIS Dashboard §Reduced ODRs, suspensions, and expulsions. § Identified a process to drill down and look at student groups to better understand the issue of disproportionality § Improved systems of proactive supports for students through prevention and education

Answering Evaluation Questions §How can HUSD reduce discipline referrals, suspensions, and expulsions? § Reduced ODRs, suspensions, and expulsions by implementing PBIS with fidelity at Tiers 1, 2, 3 §How can the district improve systems to promote environments that are conducive to learning? § Worked with the student information system Aeries to create a PBIS Dashboard § Identified a process to drill down and look at student groups to better understand the issue of disproportionality § Improved systems of proactive supports for students through prevention and education §How can HUSD improve school climate, safety, and connectedness to improve academic achievement?

Program Evaluation Reporting AUDIENCE, BENEFITS AND CHALLENGES

Audience

Benefits of Evaluation and Reporting DISTRICT LEVEL §Accountability §Concise mechanisms for sharing project outcomes to garner political support §Progress monitoring and identifying needs for refinement STUDENT OUTCOMES

Challenges of Evaluation and Reporting §Time §Change in district leadership and direction §Administrative involvement and support §Consistent use of evaluation tools §Inconsistencies between state and federal reporting requirements §Absence of data correlating academic and behavior outcomes §Competing with other evaluation priorities

Q&A >General questions >Guiding questions 1. What tools do you use to evaluate implementation? 2. How do you report evaluation results? 3. How do you address challenges to evaluation and reporting?

Resources §PBIS Program Evaluation Examples https: //www. pbis. org/evaluation-examples §Evaluation within PBIS Implementation Practice Brief http: //www. pbis. org/presentations/chicago-forum-16 §Evaluation Blueprint for SWPBS http: //www. pbis. org/blueprintguidestools/blueprint/evaluation-blueprint §PBIS APPS-Tiered Fidelity inventory https: //www. pbisapps. org/Applications/Pages/PBIS-Assessment-Surveys. aspx#tfi §DCA Resource http: //nirn. fpg. unc. edu/sites/nirn. fpg. unc. edu/files/resources/NIRN-DCA-Brief-0522 -2017. pdf

Contact Information Jake Olsen, California State University Long Beach jacob. olsen@csulb. edu Aimee Clark, Hemet Unified School District aclark@hemetusd. org Lynn Nord, Hemet Unified School District lnord@hemetusd. org

National PBIS Leadership Forum | October 4 -5, 2018, Chicago, IL Please Complete the Session Evaluation to Tell Us What You Thought of This Session