Evaluating Practices Workshop Series Session 1 What are

Evaluating Practices Workshop Series Session 1: What are we measuring and why? February 1, 2018

Welcome! Who is with us? Part C and Part B staff from over 40 states! 2

Intended Outcomes of this Session • Understand the difference between evaluating practice change and evaluating practice fidelity. • Understand what good practice change and practice fidelity outcomes, evaluation questions, and performance indicators look like. • Understand how to align intended outcomes, evaluation questions, and their related performance indicators. • Identify improvements needed to state evaluation plans for measuring both practice change and practice fidelity. 3

Working Session Agenda • Ground Rules • Summary of Pre-work • Presentation – Evaluating Practice Change and Practice Fidelity • Large Group Activity – Aligning outcomes & performance indicators to activities and evaluation questions • Small Group Work – – Developing practice change and practice fidelity outcomes and performance indicators – State identification of improvements/changes needed • Wrap-Up & Plan for Session 2 4

Ground Rules • • • Share openly (we are not recording) What is shared in the room, stays in the room Listen actively Ask one another questions Participate to the fullest of your ability – Participate in all 3 sessions – Come prepared – complete pre-work • Let us know your needs 5

POLL EVERYWHERE: Word Cloud • Instructions – Go to pollev. com/dasy on your computer, smart phone or tablet – OR text “DASY” to 22333 – Once you have joined, text your one word response to the question • Question: In one word, what comes to mind when you think about evaluating practice change and practice fidelity? 6

7

Summary of Pre-work • Most state teams reported measuring practice fidelity. • Many reported measuring practice change. • There is variation in the ways states are approaching measuring practice change and practice fidelity. • Many states have collected data on practice change/fidelity. 8

Practice Implementation Improves Results Implementation of Effective Practices Increase quantity, (e. g. , scaling up, more practices) Increase quality 9 Good outcomes for children with disabilities and their families Practice quality sustained over time

Definition of Practices • The teachable and doable behaviors that practitioners use with children and families which can be used, replicated, and measured for fidelity • The practice needs to be clearly defined in order to be measured. 10

Definitions • Practice Change: Increase or decrease in the number, frequency, precision, or quality of practices implemented by a practitioner as compared across at least two points in time. • Fidelity: The extent to which practitioners are implementing an evidence-based program or practice as intended, as measured against a threshold or predetermined level. Fidelity implies strict and continuing faithfulness to the original innovation or practice that is expected to lead to the desired child/family outcomes. • . 11

Relationship between Practice Change and Fidelity • Both practice change and practice fidelity are about the implementation of evidence-based practices (EBPs) – Are service providers changing their implementation of EBPs? – Are service providers implementing EBPs as intended? • Service providers can demonstrate changes in practices without reaching fidelity. 12

Measuring Practice Change and Fidelity • Use the same tools or methods to evaluate fidelity versus practice change. • Practice change can be evaluated in relation to fidelity. 13

Why measure practice change and practice fidelity? • To ensure practices are implemented as such that improved outcomes are expected • To determine whether practitioners maintain fidelity over time • To obtain information for monitoring progress – Incremental increases in fidelity can indicate that improvement strategies are working and highlight areas where practitioners need additional support – Corrections/adjustments in practice can be made in a timely manner 14

Evaluation Plan Components • • Outcomes Evaluation Questions Performance Indicators Measurement/Data Collection Methods 15

Example Evaluation Plan How Will We Know the Intended Outcome Description Evaluation Questions Outcome Was Achieved? Type of Outcome (performance indicator) Practitioners who receive Do practitioners who 80% of providers who receive coaching in implementing receive coaching increase their use of relationship-based implementation of the relationship-based practices intervention practices relationship-based from the checklist 6 months after Intermediateincrease use of those practices with families training when working with Practice Change practices with families and infants/toddlers with children and families. infants/toddlers with social emotional needs? social emotional needs Intermediate. Practice Fidelity Practitioners implement relationship-based intervention practices with fidelity to support families in achieving goals for the child and the family Do practitioners who 80% of practitioners implement receive coaching the targeted practices from the implement practices with checklist with fidelity? 16

Alignment Considerations of Outcomes, Questions, and Performance Indicators Outcomes • Describe what you intend to achieve as a result of activity(ies) related to EBPs • Often interconnected • Define steps toward achieving Si. MR Evaluation Questions Performance Indicators • Describe what you need to know to determine if you have achieved the outcome • Describe how you will answer your evaluation question • Are based on measurement data (e. g. , extant or new) 17

Large Group Activity: Alignment Outcome Description Evaluation Question Performance Indicator Providers use evidencebased practices to support social-emotional skills with parents and infants/toddlers identified with social emotional needs Do providers know how to use evidence-based practices to support socialemotional skills with parents and infants/toddlers identified with social-emotional needs? 80% of providers completed training on evidence-based practices to support social emotional skills with parents and infants/toddlers identified with social-emotional needs 18

Large Group Activity: Alignment Outcome Description Evaluation Question Performance Indicator Providers use evidencebased practices to support social-emotional skills with parents and infants/toddlers identified with social emotional needs Do providers use evidencebased practices to support social-emotional skills with parents and infants/toddlers identified with social-emotional needs? 80% of providers use evidence-based practices to support social emotional skills with parents and infants/toddlers identified with social-emotional needs 19

Key Take-Away Points • Practices = behaviors • Alignment, alignment • Evaluate both practice change and practice fidelity using the same measures • Evaluate practices across children, settings, families, programs 20

Small Breakout Group Work • Introductions • Structured Activity: Identify practice fidelity evaluation question and performance indicator based on a practice fidelity outcome • State Sharing: – Identify if practice change and/or practice fidelity outcomes, questions and performance indicators need to be added or modified – Identify if evaluation plan alignment is needed – Share what supports are needed in making these changes 21

Breakout Group Instructions • We will go into virtual breakout groups in Adobe Connect • For visual – you will be automatically moved into breakouts through this Adobe Connect room • For audio – you will call into conference line for each specific breakout group (next slide & in Adobe note) 22

Small Group Agenda • Who’s with us in the small group? • Structured Activity: Identify practice fidelity evaluation question and performance indicator based on a practice fidelity outcome • State Sharing: – Identify if practice change and/or practice fidelity outcomes, questions and performance indicators need to be added or modified – Identify if evaluation plan alignment is needed. – Share what supports are needed in making these changes 23

Practice Fidelity Outcomes/Measures Outcome Description Evaluation Questions Practitioners demonstrate fidelity in their use of evidence based practices to support social emotional skills 24 Performance Indicator

Evaluation Plan Worksheet Handout 25

State Sharing Re: Evaluation Plan Improvements • What do you need to add or modify in your evaluation plan to effectively evaluate practice change and practice fidelity and ensure alignment (e. g. add an outcome, revise a performance indicator)? • What supports do you need in making these changes to your evaluation plan? 26

Your Take Aways • What struck you today? • What did this get you thinking about? 27

Formative Evaluation • What worked well? • What could be improved for next time? 28

Wrap-Up • Session 2: February 15 th from 3: 30 -5 pm Eastern – How are we measuring? Delving deeper into measurement strategies and data sources • Pre-work - will be sent out next week – Defining and operationalizing practices so they are measurable – Characteristics of a high-quality tool for measuring practice change and fidelity 29

Thank you See you in 2 weeks! The contents of this presentation were developed under grants from the U. S. Department of Education, # H 373 Z 120002, #H 326 P 120002, H 326 R 140006, and H 373 Y 130002. However, those contents do not necessarily represent the policy of the U. S. Department of Education, and you should not assume endorsement by the Federal Government. Project Officers: Meredith Miceli, Richelle Davis, Julia Martin Eile, Perry Williams, and Shedeh Hajghassemali. 30