Oregon ScalingUp WELCOME OREGON SCALINGUP EBISS UTILIZING IMPLEMENTATION

Oregon Scaling-Up WELCOME OREGON SCALING-UP EBISS UTILIZING IMPLEMENTATION SCIENCE FRAMEWORKS TO ENSURE EFFECTIVE BEHAVIORAL & INSTRUCTIONAL SUPPORT SYSTEMS (EBISS) SPRING 2016 PENDLETON, VALE AND REDMOND OREGON

INTRODUCTIONS MARICK TEDESCO, PH. D. STATE TRANSFORMATION SPECIALIST SARA FALCON, PH. D. OREGON EBISS INITIATIVE

GOALS FOR TODAY 1. PROVIDE A REVIEW OF THE CRITICAL FEATURES OF EFFECTIVE BEHAVIORAL AND INSTRUCTIONAL SUPPORT SYSTEMS (EBISS) 2. INTRODUCE KEY CONCEPTS OF IMPLEMENTATION SCIENCE (IS) 3. DEVELOP UNDERSTANDING OF THE FRAMEWORKS ASSOCIATED WITH IS 4. IDENTIFY NEXT STEPS FOR YOUR TARGET INITIATIVE

INTRODUCTIONS DISTRICTS District Number of team members Team member roles

AGENDA 8: 00 – 8: 30 – 9: 00 – 10: 15 – 10: 30 – 11: 30 – 12: 00 – 1: 00 – 2: 15 – 3: 45 Registration and Pre Test Introductions, Data requirements review Competency Drivers – Staff Selection Break Organization Drivers Improvement Cycles Lunch Terms of Reference Post Test and Evaluation Team Time

ACTIVE IMPLEMENTATION FRAMEWORKS WHO WHAT Teams Intervention WHEN Stages HOW Cycles Drivers

IMPROVED EDUCATIONAL OUTCOMES Consistent Use of Educational Innovations Interventions meet Implementation Performance Assessment (Fidelity) Systems Intervention er riv y D nc ete s Co er riv mp D Selection Facilitative Administration on ati Training niz ga Or s Coaching Decision Support Data System Leadership Adaptive Technical © Fixsen & Blase, 2008

THERE ARE 3 CATEGORIES OF IMPLEMENTATION DRIVERS: Competency Drivers – are mechanisms to develop, improve and sustain one’s ability to implement an intervention as intended in order to benefit students. Organization Drivers – are mechanisms to create and sustain hospitable organizational and system environments for effective educational services Leadership Driver – focus on providing the right leadership strategies for different types of leadership challenges. These leadership challenges often emerge as part of the change management process needed to make decisions, provide guidance, and support organization functioning.

IMPLEMENTATION DRIVERS COMMON FEATURES OF SUCCESSFUL SUPPORTS TO HELP MAKE FULL AND EFFECTIVE USE OF A WIDE VARIETY OF INNOVATIONS

Positive Outcomes for Students rs e riv er s riv nc y D ete D Core Implementation Components on ati niz ga Or Professional Development/ Professional Learning How: Effective Educational Practices mp What: Co Why: Leadership Capacity to provide direction and vision © Fixsen & Blase, 2008

Implementation Drivers Performance Assessment (Fidelity) y D riv e rs Coaching Selection Co mp ete nc Training © Fixsen & Blase, 2008

Competency Driver STAFF SELECTION Selection of staff, from an Active Implementation perspective is different from “selection as usual” in two important ways: • Mutual Selection • Behavioral Rehersals and Role Play components of the interview process

Competency Driver STAFF SELECTION • Selection is viewed as a “mutual” process. • For example, a school or district is deciding whether or not to select an individual to join them and the process allows the applicant to understand the expectations related to the position in order to determine if their skills abilities and attitudes are a good “fit”.

Competency Driver STAFF SELECTION Selection includes “role play” or “behavior rehearsal” processes. These processes allow interviewers to observe how applicants respond to feedback and how able and willing they are to learn new practices. This process also provides insight into how an applicant might respond to feedback and data. A desire and ability to learn and grow are critical for ongoing improvement.

Competency Driver STAFF SELECTION Think about your recruitment, interviewing, and selection process. After reviewing the two key distinctions discussed in the slides above, what are two things you could do to improve the Selection process (including recruitment, interviewing, and selection) in your setting so that it better supports effective implementation?

SELECTION Co m pe te nc y. D riv er s Implementation Drivers Purposes • Select for the “tough to teach traits” • Screen for pre-requisites • Set expectations for new hires – use of data, coaching • Allow for mutual selection • Improve likelihood of retention after “investment” • Improve likelihood that training, coaching, and supervision will result in implementation

SELECTION DRIVER CHALLENGES Co m pe te nc y. D riv er s Implementation Drivers “We have who we have …this doesn’t apply to us!” What’s the way forward?

INTEGRATED AND COMPENSATORY IMPLEMENTATION DRIVERS • Integrated Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI) • Compensatory er riv y D te nc pe Co m Integrated & Compensatory s er riv n D tio za ni ga Or s • At the practitioner level • At the program level Leadership Drivers

BENEFITS OF DRIVER-BASED ACTION PLANNING Implementation Drivers Action Plans • Infrastructure needed becomes visible to all • Strengths and progress get celebrated • Next right steps are planned and results measured • Resources can be aligned and re-purposed to improve implementation

NEXT STEPS • Locate the Staff Selection activity to review job description and interview samples. For the purpose of this module, consider that these represent selection practices from a single district. • Utilize the Staff Selection best practices self-assessment to rate the samples discussed above. • Develop two action items to address a category that needs improvement.

ORGANIZATION DRIVERS Implementation Drivers n tio iza an g Or • Change Organizations and Systems • Create and sustain hospitable organizational and system environments for effective services • Develop functional data systems that can be used to inform decision-making rs ive Dr

All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get. “ —R. Spencer Darling Business Expert ORGANIZATIONAL CHANGE

SYSTEM CHANGE EXISTING SYSTEM EFFECTIVE INNOVATIONS ARE CHANGED TO FIT THE SYSTEM EXISTING SYSTEM IS CHANGED TO SUPPORT THE EFFECTIVENESS OF THE INNOVATION EFFECTIVE INNOVATION

SYSTEM CHANGE Creating Capacity for Competent Change • New innovations do not fare well in current organizational structures and systems • Develop new position descriptions and job functions in State Departments of Education and in Regional and District systems. “Systems trump programs. ” —Patrick Mc. Carthy, Annie E. Casey Foundation

IMPLEMENTATION DRIVERS Performance Assessment (Fidelity) s riv y D nc ete mp er riv Co Facilitative Administration D on ati Selection Systems Intervention niz Training ga Or er s Coaching Decision Support Data System © Fixsen & Blase, 2008

DECISION SUPPORT DATA SYSTEMS Implementation Drivers Purposes ni ga Or • Monitor and improve student outcomes through data-based decisions • Provide information to assess effectiveness of intervention and prevention practices • Analyze the relationship of fidelity to outcomes • Guide further program development n tio za rs ive Dr • Detect discrete issues as well as systemic issues • Engage in continuous quality improvement • Of the Intervention and the Drivers • Celebrate success • Be accountable to parents, Board of Education taxpayers, and other funders

IMPLEMENTATION DRIVERS Performance Assessment (Fidelity) s riv y D nc ete mp er riv Co Facilitative Administration D on ati Selection Systems Intervention niz Training ga Or er s Coaching Decision Support Data System © Fixsen & Blase, 2008

Implementation Drivers FACILITATIVE ADMINISTRATION Purposes n tio iza an g Or rs ive Dr • Creates an internally hospitable environment for the new way of work – at the level of the “agency” (e. g. school, District) • Facilitates the installation, implementation, and improvement of the Drivers for each innovation • Takes the lead on Systems Interventions • Looks for ways to make the direct work of practitioners (e. g. teachers, school staff) and administrators more effective and less “burdensome”!!

IMPLEMENTATION DRIVERS Performance Assessment (Fidelity) s riv y D nc ete mp er riv Co Facilitative Administration D on ati Selection Systems Intervention niz Training ga Or er s Coaching Decision Support Data System © Fixsen & Blase, 2008

SYSTEMS INTERVENTION Implementation Drivers Purposes n tio iza an g Or rs ive Dr • Identify and “lift up” systemic barriers and facilitators to the next level to improve support for the new way of work • Create an externally “hospitable” environment for the new way of work • Embed facilitators and strengths • Contribute to cumulative learning in multi-site projects

CHALLENGES Organizational Drivers Data Systems that. . n tio iza an g Or • Warehouse the data and are not timely • Produce reports that are not provided at the actionable unit • And… rs ive Dr Site-based administration that is not responsive to feedback Systems that block feedback or do not respond to it

BIG IDEAS THAT HELP Organizational Drivers • Linked Teaming Structures • Linking Communication Protocols (e. g. agenda links, meetings dedicated to feedback) n tio iza an g Or s er iv Dr • Terms of Reference • Practice to Policy Feedback surveys and action plans

IMPLEMENTATION DRIVERS Performance Assessment (Fidelity) riv y D Facilitative Administration s Co mp er riv D ete on nc ati Selection Systems Intervention niz Training ga Or er s Coaching Leadership Adaptive Decision Support Data System © Fixsen & Blase, 2008 Technical

MANAGING THE FLOW Technical Adaptive

DRIVERS INTEGRATION WITH OTHER FRAMEWORKS Implementation Drivers Teams • Locus of Responsibility • Monitor Integration Cycles • Within a Driver • Among Drivers • Feed Forward • Feedback Stages • • Exploration Installation Initial Implementation Full Implementation

INTEGRATED AND COMPENSATORY IMPLEMENTATION DRIVERS • Integrated Consistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI) • Compensatory er riv y D te nc pe Co m Integrated & Compensatory s er riv n D tio za ni ga Or s • At the practitioner level • At the program level Leadership Drivers

BENEFITS OF DRIVER-BASED ACTION PLANNING Implementation Drivers Action Plans • Infrastructure needed becomes visible to all • Strengths and progress get celebrated • Next right steps are planned and results measured • Resources can be aligned and re-purposed to improve implementation

NEXT STEP • Utilize the Organization Drivers Mapping Tool with your team to determine next steps for assessment and action planning.

Active Implementation What about Improvement Cycles? HOW Cycles

WHAT HAPPENS PRIOR TO IMPLEMENTING A PDSA CYCLE? 1. Team convenes to decide on a change objective 2. Team works to understand the problem 3. Team brainstorms a number of possible changes 4. Possibilities are prioritized and one is selected for testing 5. The test occurs in one portion of the agency 40

PDSA Cycles: Trial & Learning Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990) • Plan – Decide what to do • Do – Do it (be sure you did it as intended) • Study – Look at the results • Act – Make adjustments • Cycle – Do over and over again until the intended benefits are realized

IMPROVEMENT CYCLE USES • Rapid Cycle • For a Particular Problem • Practice Improvement • Usability Testing • Practice-Policy Feedback Loops • Transformation Zones

RAPID CYCLE • Problem-solving • Team Lead identified • Right people on the team • Time-limited to address the problem • “Solves” the problem • Team disbands

RAPID CYCLE • Practice Improvement • Relatively short-cycle • On-going efforts to improve practices and competencies • Define the Problem • Define Metrics • Measure • Intervene • Measure Again • Use data to achieve better outcomes and “embed” solutions

IMPROVEMENT CYCLE USES • Rapid Cycle • For a Particular Problem • Practice Improvement • Usability Testing • Practice-Policy Feedback Loops • Transformation Zones

PURPOSES • The overarching purposes of usability testing are to improve and better operationalize • “Challenging” intervention components. • The Implementation Drivers (especially competency drivers) WHY IS IT HELPFUL? WHAT’S THE PURPOSE? • Fidelity and outcome data collection processes • So that major ‘bugs’ are worked out of the processes and the processes can be effective 47

USABILITY TESTING PROCESS • 4 or 5 “typical users” to try out the current version of the implementation methods and/or the innovation practices • Fix the problems from the first “trial” • Repeat with a new group of 4 or 5 typical users • Fix the problems form the second “trial” • After 4 or 5 “trials” you should have most of the problems solved

RAPID-CYCLE TESTING Rapid-Cycle changes Ø Quick experiments; do-able within 2 weeks PDSA cycles Ø Plan the change Ø Do the plan Ø Study the results Ø Act on the new knowledge 49

PLAN • Define the change behaviorally… precisely what will be done? • Who will implement the change? • What preparation needs to be done before starting the change? • Clarify who will measure the change and who will review the data regularly to share with the team. 50

DO • Note the start date for the cycle • How long will the initial test last? • How often will the team meet to assess progress and review data? • During the test, the team considers what improvements might need to be made to improve the results 51

RAPID-CYCLE TESTING - EXAMPLE Rapid-Cycle change Ø “Barrier question” PDSA cycle with 2 Principals PDSA cycles Ø Plan: Design question Ø Do: 2 schools use on 1 grade level each Ø Study: 70% of teachers respond positively to 2 postobservation sessions compared to 50% Ø Act: Try with another 8 -10 teachers 8 52

STUDY • After the agreed upon test period the team reviews the results • Team leader assembles and graphs the data for the team • The team deliberates what it has learned • Based on the learning, the team considers whether a change in strategy is in order 53

ACT • In light of what has been learned, the team decides what to do next • Make an adjustment in the strategy to get closer to the objective • Increase the objective if the initial target has been met and the team believes even more progress can be made • Combine another change with the existing strategy to start a new cycle • Abort the existing strategy and start a new cycle 54

IT IS IMPORTANT TO REMEMBER… • It often takes more than one cycle to achieve your objective. • By changing only 1 thing at a time you know the impact of your change. • Sometimes several changes are necessary to maximize the improvement you seek. 55

CHANGE CYCLES A P TA DA A S Hunches Theories Ideas S D D S P A P Changes That Result in Improvement Cycle 4 Cycle 3 D A P Cycle 2 S D Cycle 1 56

KEYS TO PDSA SUCCESS 1. Be clear about your target objective 2. Implement only 1 change at a time 3. Make sure all participants are implementing the change as planned 4. Implement the change in a small portion of the agency 5. Study the results data before making a modification to the plan 6. Do not hesitate to start a new cycle when the data indicates or the team is convinced that a modification will improve results 57

AND LAST… Keep measuring and studying the results until you reach your objective and determine you have improved as much as you can 58

ACTIVITY • Utilize the PDSA Planning Form with your team to identify and document a plan for initiating a component of your targeted initiative. 59

IMPLEMENTATION STAGES Outline the integrated, non‐linear process of deciding to use an effective intervention and finally having it fully in place to realize the promised outcomes. Active implementation stages are Exploration, Installation, Initial Implementation and Full Implementation.

TERMS OF REFERENCE • Terms of Reference: A Tool and Process for Clarity and Communication 61

SETTING THE STAGE FOR MULTI-LEVEL GROUP WORK • Establishing Terms of Reference (To. R) • Terms of Reference (To. R) are detailed in a written document • Outlining the purpose of the project or group, • How it will be structured and implemented. • It is like an internal Memorandum of Understanding (MOU) • Helps you clarify communication, mandates, and core features and functions of the group 62

TERMS OF REFERENCE • To. R is quite frequently used in business and planning endeavors • Creating a detailed To. R is a critical part of any project • The To. R is an important Plan step in a PDSA Cycle for Group Work Plan Act Do Study 63

BENEFITS • To. R can facilitate work because: • It establishes common ground a common understanding of why a group has been formed and what is expected. • It is a proactive way to ensure that there is agreement about important dimensions of the project. • It clarifies both the role of the team and may also clarify individual roles on the team • Increases comfort and legitimacy of “all voices” on the team • Such agreement, in the absence of work or issues, helps the group talk more openly about diverse opinions and reach consensus. 64

TOR FOR AGREEMENT CREATION: COMMON VALUES • Details “values and ways of work” that the group will abide by/use/review throughout the project • An opportunity – in the absence of the actual work to: • Surface Challenges • Get confusion on the table and move to clarity • Reduce the likelihood of ‘distractions’ during the work of the group 65

KEY TOR COMPONENTS • • • Vision for the Work Together Scope and Objectives Expected Outcomes and Deliverables Boundaries (what it is and is not, when are we done) and Limitations Authority, Accountability and Reporting Requirements • Linking Communication Protocols for Alignment (with whom do we communicate, how, and how often and for what purpose) Roles and Functions of individuals (who participates in what ways) • Leadership • Term • Membership • Orientation for New Members Resources available to the project Decision-making process Values and Ways of Work Implementation Plans 66

Plan Act USE IT OR LOSE IT (DO) Do Study • If the document is actively used and referenced throughout the life of the project it can: • Help maintain focus by: • Using the To. R so that the project or group remains on track or on “mission” • Serving as an aid to troubleshooting to maintain focus (e. g. “are we still aligned on values and way of work? ”, “do we have the authority, people, resources we need right now? ”) • Serve as a ‘prompt’ before a meeting, a discussion, or a decision. 67

Plan USE IT OR LOSE IT (DO) Do Act Study • If the document is actively used and referenced throughout the life of the project it can: • Help refocus when group dynamics are bumpy (e. g. “I wonder if we should revisit our agreed upon way of work, we said…. . but now I am hearing…, So have we changed our way of work or scope of work or have we strayed from our roles/responsibilities, values? ”). • Support “getting on the balcony” with respect to group process, dynamics, and agreement creation – “let’s think about our end goals” 68

REVISIT, REFLECT, REVISE Plan Do Act • Nothing is carved in stone Study • Revisit and Reflect (Study) In the absence of actual issues - Get on the Balcony Identify emerging challenges • The very “work of the group” may call for a change in the To. R • Opportunity to be sure all are heard • Revise (Act) Change any elements to improve group functioning or the success of the overall project Go again (PDSA) 69

NEXT RIGHT STEPS • Getting Started • How might you go about introducing the idea of Terms of Reference to a group or project team that is just forming (e. g. the Team’s role: New functions for District and School Leadership Teams)? • What “case” could you make? • Who would you talk with before ‘making the case’? • Getting Re-started • Is a current group, committee, or project “stuck” or wrestling with adaptive challenges? • Could they benefit from “re-starting” by developing a Terms of Reference document? • How and with whom might you begin? 70

ACTIVITY • Team members to brainstorm a list of the characteristics of the ineffective meetings they’ve attended, and write it on chart paper. Then ask them to list the characteristics of effective meetings. • Using the lists as a basis, ask team members to volunteer ideas for specific agreements needed to ensure effective team functioning. You may wish to suggest topics to address (e. g. , attendance, agenda, minutes, meeting tasks, tasks between meetings, decision-making, handling conflict). • Coach the members who introduce ideas to say precisely what they need from the group, and how agreement about it will promote individuals’ comfort and the team’s effectiveness. If no one says any- thing contradictory, you can accept all these needs as group agreements. 71

RESOURCE • Complete the Developing Your Terms of Reference (TOR) worksheet with your team. 72

IMPLEMENTATION MATH Effective Interventions Effective Implementation (The “WHAT”) (The “HOW”) Positive Outcomes for Students (The “WHY”)

PLAN FOR CHANGE • It is not a “school problem” • District is the point of entry for sustainable school improvement • Use short-term infusion of resources • Establish long-term, district-based capacity for quality

75

IMPLEMENTATION DRIVERS ARE DYNAMIC! A key feature of Implementation Drivers is their integrated and compensatory nature: Integration – means that the philosophy, goals, knowledge and skills related to the program or practice are consistently and thoughtfully expressed in each of the Implementation Drivers. Compensatory – means that the skills and abilities not acquired or supported through one driver can be compensated for by the use of another driver.

IMPROVED EDUCATIONAL OUTCOMES Consistent Use of Educational Innovations Interventions meet Implementation Performance Assessment (Fidelity) Systems Intervention er riv y D nc ete mp Co s er riv D Integrated & Compensatory Technical Facilitative Administration on ati Adaptive niz Selection ga Training Or s Coaching Decision Support Data System Leadership © Fixsen & Blase, 2008

NEXT STEPS AND TIMELINES 78

TEAM TIME ACTIVITY: ACTION PLANNING CONTINUED 79