DSFI Overview Stakeholder Engagement Funding and Alignment Policy

DSFI Overview

Stakeholder Engagement Funding and Alignment Policy Workforce Capacity Executive Functions LEADERSHIP TEAMING Implementation Functions Training Coaching Local Implementation Demonstrations Evaluation

DSFI Purpose To guide District Leadership Teams in the assessment, development, and execution of action plans that promote the capacity for sustainable, culturally and contextually relevant, and high-fidelity implementation of multi-tiered social, emotional, and behavioral systems of support and practices.

DSFI Logistic Who: Members of a district’s planning team, with the active presence and guidance of an external state or regional facilitator When: Annually Currently available on field test: Your state coordinator can give you access!

Guidelines 1. Identify an internal or external facilitator who is fluent in implementation of a tiered behavioral system, the critical features of the DSFI, and data for action planning. 2. Form a District Leadership Team, composed of personnel who have been identified to establish and lead implementation of the tiered behavioral system. 3. Gather and review existing district data related to: • student behavior • organizational knowledge • program implementation fidelity

Guidelines 4. 5. 6. 7. 8. 9. A consensus approach is used to complete the DSFI. *Note the existence of the glossary to help clarify items as needed Assessment results are used to inform district planning. Review implementation elements to develop steps for a multi-year action plan (1 year, 2 -4 years, and 5+ years). To build sustainability and internal capacity, emphasize use and organization of existing resources for implementation of action plan. As a team, review progress toward action plan activities and outcomes on a regular basis (e. g. , monthly or quarterly) Conduct annual evaluation, develop action plan, monitor implementation and outcomes.

Developing Effective Evaluation

Stakeholder Engagement Funding and Alignment Policy Workforce Capacity Executive Functions LEADERSHIP TEAMING Implementation Functions Training Coaching Local Implementation Demonstrations Evaluation

Evaluation • The act of measuring outcomes and comparing them to initial goals to assess: – Whether goals were met – Whether to: • Continue • Adjust • Abandon Evaluation

SCTG LEA GPRAs The number of training and/or technical assistance events to support implementation with fidelity provided annually by LEAs to schools implementing a multi-tiered system of support. Number and percentage of schools annually that report an improved school climate based on the results of the EDSCLS or similar tool. ! r e d r a H t o N r Number and percentage of schools annually that are implementing a multi-tiered e t ar m S k r o W system of support framework with fidelity. o s l a l l i w n o Number and percentage of schools annually that are implementing opioid abuse i t a u l a v e A R d P o o G g prevention and mitigation strategies. r u A o y l l a s s e r d d a r e w s Number and percentage of schools that report an annual decrease in suspensions an and expulsions related to possession or use of alcohol. Number and percentage of schools that report an annual decrease in suspensions and expulsions related to possession or use of other drugs.

Steps for Effective Evaluation 1. Identify the Purpose – 2. 3. 4. 5. 6. 7. 8. Summative, Formative, Accountability Identify Key Stakeholders Select an Evaluation Team Define the timeline Define Core Evaluation Questions Select Evaluation Measures Conduct the Evaluation Report and Use the Findings See Evaluation Blueprint for more detail

Categories for Core Evaluation Questions Organizing Reach Process Capacity Fidelity Outcome To what extent are schools adhering to the critical features of a What is the LEA capacity for sustaining and expanding this work? What are the key activities associated with your SCTG? What is the purpose of this evaluation? Who is participating in your SCTG? What are the differences for students and staff as a result of your SCTG? program or practice?

Organizing • What is the purpose of this evaluation? • Who are the key stakeholders? • What are the goals and objectives of implementing PBIS? • What are the decisions that will be made based on the evaluation?

Defining Core Evaluation Questions Evaluation Question Reach Process Fidelity Capacity Outcomes Measures Metric/Data Produced Collection Cycle

Reach: Who is participating in your SCTG? Sample Evaluation Question What schools are receiving support (i. e. , in the initiative)? How has MTSS sustained over time?

Reach: Who is participating in your SCTG? Evaluation Question Measures What schools School Profile Form are receiving Schools Implementing support (i. e. , in (PBIS Assessment/Eval) the initiative)? How has MTSS Schools Implementing sustained over (PBIS Assessment/Eval) time? Metric/Data Produced Collection Cycle Name, address, contact, enrollment, grade levels, demographics, start year Number of schools reporting fidelity per year When a school begins implementation Updated annually

Process: What are the key activities associated with your SCTG? Sample Evaluation Question What steps are the Leadership Team taking? Who in the district is providing training and Coaching? How are district coaches supported? What professional development has been delivered? What coaching has been delivered? Was professional development perceived as useful? How can the MTSS initiative be improved according to stakeholders?

Process: What are the key activities associated with your SCTG? Evaluation Question What steps are the Leadership Team taking? Who in the district is providing training? Who in the district is providing coaching? How are district coaches supported? Measures Metric/Data Produced DSFI Action Plan Items Report Number of action plan Completed annually goals met Trainer List of trainers Updated annually Coaches List of Coaches Updated annually Coaches’ Survey List of coaches’ meetings and trainings provided Used to identify training Collected annually needs Collected after each Number of events and training event coaches present at each Collection Cycle

Process: What are the key activities associated with your SCTG? Evaluation Question What professional development has been delivered? What coaching has been delivered? Was professional development perceived as useful? How can the MTSS initiative be improved according to stakeholders? Metric/Data ng. Produced i n i a e c tr f n o a t r List of events, training mbe ical assis. Number of events u n The r techn rt and teams present at ity content, and school l o o e / p d d i p f an su itheach o t to w s s t n teams participating A n o i E e ev ntat ally by L e lti u m e m l u Number of events Coaching log, content, imp ed ann ng a. i t n e d i t prov ls implem f and teams present at ppor and school teams u s hoo ystem o each c s participatingred s -tie Workshop Evaluation % reporting that the event was useful Form Measures Collection Cycle Collected after each PD event Collected after each coaching event Collected after each PD event Stakeholder Input and % satisfaction Collected in each Satisfaction Surveys school annually (SISS) – Family, School Descriptive feedback Personnel, Student

Capacity: What is the LEA capacity for sustaining and expanding this work? Sample Evaluation Question What resources are available to support MTSS? What capacity of each district to support MTSS?

Capacity: What is the LEA capacity for sustaining and expanding this work? Evaluation Question Measures Metric/Data Produced Collection Cycle What resources are Organizational FTE List of resources Updated annually available to support Budget information (individuals, funding sources) MTSS? Percent of What capacity of District Systems Completed annually each district to Fidelity Inventory implementation support MTSS? (DSFI)

Fidelity: To what extent are schools adhering to the critical features of a program or practice? Sample Evaluation Question What is the fidelity of implementation of MTSS at each school? What is the fidelity of implementation of related practices at each school? What schools can serve as local implementation demonstrations (across tiers and levels)?

Fidelity: To what extent are schools adhering to the critical features of a program or practice? Evaluation Question Measures of e g a t en What is the fidelity of d perc. Tiered Fidelity re a t n a a d th er e y b l r l implementation of Inventory (TFI) e m a i t Nu annu a multi- work s l o MTSS at each school? me scho enting - Tiers 1, 2, 3 a r f t m or imple of supp m syste delity. What is the fidelity of Lesson calendar fi of e h g t i a w ent e c r e implementation of and check-off nd p that ar e a r e b related practices at ually oid abus Num ls ann. Observation of o opi gation g n i each school? lessons t scho men and miti e l p im n o i t n e prev gie. Tiered Fidelity s. What schools can serve e t stra as local implementation Inventory (TFI) demonstrations (across - Tiers 1, 2, 3 tiers and levels)? School Profile Form Metric/Data Produced % of schools implementing each Tier with fidelity (criterion: 70%) Collection Cycle % of lessons taught on time (goal: 100%) % of lesson’s critical features observed (goal: 90%) List of schools and contact information Brief case studies for evaluation report Submitted annually by each school team Annually Progress monitoring as needed Updated annually

Outcome: What are the differences for students and staff as a result of your SCTG Sample Evaluation Question If MTSS is implemented to criterion, is there improvement in social & academic outcomes for students? Discipline rates Office referrals, Suspensions/expulsions, restraint and seclusion Academic and related outcomes Achievement, Attendance, Graduation, Career and college readiness Social-emotional-behavioral outcomes Social and emotional competencies and skills, Mental health outcomes Substance use/abuse Special education decisions Equity in each of the above outcomes Do schools implementing MTSS to criterion have more positive school climate? What other outcomes are associated with MTSS implementation? Adult perceptions, Staffing , Educational/behavioral policy

Outcome: What are the differences for students and staff as a result of your SCTG Evaluation Question Measures Metric/Data Produced nd a ols r o e h b c s Num tage of d al s n u l a n o n r n o e % reductions in: e a b If MTSS is f sch al perc port an o Num School Outcome e g nta nnu - Office Disc. Referrals at re in a h implemented to perce. Data Summary t n a ort ase p e r e nd r c a t e - Suspensions s a d n criterion, is there Form h t io to in s d n e e s e t a p a e sus rel and to Student/Admin. time recouped s s f n improvement in decr - Attendance n o o o i i e s s l s d n u u pe or exp elate f % increases in: n r o i s s social & academic sus - ODRs n s o io osse rugs. p xpuls ion or use e - Attendance her d outcomes for - Suspensions s s e s s ot po. - Academics l o students? - Academics alcoh Do schools implementing MTSS to criterion have more positive school climate? School Climate Mean school climate rating Sub-scale school climate ratings Survey (SC Suite) d s l n o a o r h e umb age of sc rt an - N Student nt po e e r c r t e a p - Family y th ool l l a u ann ved sch the - School n o o r imp e based SCLS or t Personnel he ED clima of t s t l u res ol. o t r a simil Collection Cycle Collected Annually (by end of March)

Options for Measuring Outcomes • Compare schools to themselves – Compare pre vs. post implementation • Compare schools to each other – Compare schools in vs. out of the initiative – Compare high vs. low fidelity schools – Compare high fidelity schools vs. state/district averages

Steps for Effective Evaluation 1. Identify the Purpose – 2. 3. 4. 5. 6. 7. 8. Summative, Formative, Accountability Identify Key Stakeholders Select an Evaluation Team Define the timeline Define Core Evaluation Questions Select Evaluation Measures Conduct the Evaluation Report and Use the Findings See Evaluation Blueprint for more detail

Reporting and Using Evaluation Results • Create evaluation report – Report with clarity (and graphs) – Frame within larger concern – Recognize limitations • Disseminate to all stakeholders • Use results for continuous improvement – Connect fidelity with outcomes! • • If implemented poorly, what is needed to implement well? if effective, what do we need to sustain it? If ineffective, what needs to change, or should it be abandoned? How does it relate to other priorities and initiatives?

DSFI Element 8: Evaluation 8. 1 8. 2 8. 3 8. 4 8. 5 8. 6 8. 7 8. 8 8. 9 8. 10 Table Discussion (25 min) Evaluation Plan 1. Review “ 8. Evaluation” items Data Collection Systems on the DSFI. Evaluation Feedback Loop 2. Discuss your current priorities Differentiated Supports for evaluation based on (a) Assessment and Evaluation of specified grant activities and (b) Progress identified needs within your Student Identification Data LEA. Level of Use 3. Discuss available resources with Student Performance Data other teams and TA providers. Annual Evaluation 4. Identify and document 1 -2 Acknowledgment of Progress important activities on your action plan.

Share out and Questions • Share 1 -2 new action steps you plan to take • Share something you’ve done to effectively evaluate the work in your LEA • Share a resource you’ve used to evaluate one of these key categories • What questions do you have about conducting effective evaluations in the context of your SCTG

Annual Implementation Survey • We use these surveys to develop an understanding of where districts are with respect to the implementation of a multi-tiered system of support. • This is also your opportunity to provide feedback on technical assistance supports and provide suggestions for future webinar and newsletter topics as well as topics to be covered at the SCTG pre-conference workshop in Chicago next year. • Remember, this information will not be used to evaluate individual projects. It will be used to guide the planning of our technical assistance offerings as well as to share the “big story” about the overall implementation progress of grantees. https: //uconn. co 1. qualtrics. com/jfe/form/SV_b 7 x. PXRnd. UPcw. L 0 V