Vision Every child in every district receives the

Vision: Every child in every district receives the instruction that they need and deserve…every day. Data-Based Decision Making Cadre 11 Training October 2017

RTI Essential Components SLD Decision Making Progress Monitoring Data-Based Decision Making with Decision Rules Standards of Practice Interventions Leadership Teaming/Data. Based Decision Making Training Coaching Fidelity Screening Core Culture Professional Learning & Support

How does your district make decisions? SLD ELIGIBILITY Individual Students INTERVENTIONS (TIER 2/3) Groups of Students Individual Students CORE (TIER 1) District Level School Level Grade Level 3

An Important Caveat We often focus on the Data-Based Decision Making when it’s the Data-Based Decision Implementing that really matters most A decision might be “good” or “bad” depending on the quality with which it is implemented

Solve These Problems

A bat and a ball cost $1. 10 in total. The bat costs $1. 00 more than the ball. How much does the ball cost?

If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?

In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

1. A bat and a ball cost $1. 10 in total. The bat costs $1. 00 more than the ball. How much does the ball cost? 5___ cents 2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? 5___ minutes 3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, 47 the patch to how long would it take for

Cognitive Reflection Test Shane Frederick (2005) • 83% of people get at least 1 question incorrect • 33% of people get all 3 incorrect • 52% of MIT students got at least 1 incorrect Intuition vs. Deliberation 33% 17% of people

Which line is longer? Intuitions can be deceiving Which line looks longer?

Round Robin Reading Despite no research support of its effectiveness, 58% of teachers surveyed reported using Round Robin Reading, or a similar alternative. • Of teachers that used it, 30% knew it was not supported by research and still used it (Ash, Kuhn, & Walpole, 2009)

Teacher judgment of student reading abilities (Begeney, Krouse, Brown & Mann, 2011) Compared teacher judgment of student reading levels to student DIBELS scores (grades 1 -6) • “…teachers often made inaccurate judgment about students’ actual reading performance, tended to overestimate students’ abilities, and were better judges of high-performing versus low- or average-performing readers. ” • When asked to identify a student’s level of reading risk level, teachers were correct for roughly 58% of their students – There’s a 33% chance of being correct by randomly guessing (Similar findings in Begeny, Eckert, Montarello, & Storie, 2008; Feinberg & Shapiro,

Teacher Referral by Race • 75% of referrals for SPED come from teachers (Lloyd et at, 1991) • Between 73% and 90% of students referred are found eligible (Harry & Klingner, 2007) Probability of being referred for eligibility testing 1 0. 8 0. 6 0. 93 0. 88 0. 63 0. 69 0. 68 0. 59 0. 56 0. 52 0. 40 0. 4 Black Latino 0. 2 0 White Learning Disability Behavior/Emotional Disorder Talented & Gifted (Fish, 2016)

Our ability to reason is flawed Confirmation Bias - We are more likely to believe (and to remember) information that confirms our views rather than information that disagrees with it, regardless of the accuracy of the information Availability Heuristic - We’re biased towards information that is more readily available • In percentages, how much do you contribute to the upkeep of your home? – How would your spouse/significant other answer this same question?

Our ability to reason is flawed More confidence ≠ Higher likelihood of being correct “…declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true. ” -Daniel Kahneman People tend to make decisions based on stories and not on evidence

Moving from Intuition to Deliberation • When deliberation takes time and effort, intuition takes over • Intuitive errors cannot be avoided • “Learn to recognize situations when mistakes are likely and try harder to avoid significant mistakes when the stakes are higher. ” -Daniel Kahneman

“The job of the decision maker [isn’t] to be right but to figure out the odds in any decision and play them well. ” Making a decision using data and evidencebased programs/practices increases the odds of success and reduces the impact of our biases However…

However… …it does not guarantee success

Outcome Bias You’ll never truly know if you made the best decision, because you can’t go back and make a different decision and compare outcomes. It is often only in hindsight that decisions are “good” or “bad”, “right” or “wrong”

“Complexity requires group success” -Atul Gawande • Treating patients – Doctors, Nurses, Technicians, Pharmacists, Administrators • Flying a plane – Pilot, Co-pilot, Air traffic controllers, Mechanics, Flight attendants • Building a house – Engineers, Carpenters, Electricians, Plumbers, Landscapers, etc… • Teaching students – Teachers, Principals, Specialists, SPED, Instructional assistants, Parents

What does this mean for databased decision making? • Focus less on making the “right” decision, and more on making the “best” decision you can – “Wherever there is uncertainty there has got to be judgment, and wherever there is judgment there is an opportunity for human fallibility” -Don Redelmeier • The “best” decision you can make for a student is one that… …is based on all of the best data/evidence available (e. g. , is deliberate & has the best odds of success) …everyone on the team is committed to implementing well

Problem Solving Process for Data-Based Decision-Making

Why Problem Solving? Consistent Framework at all helps with • levels Efficiency • Effectiveness • Sustainability SLD ELIGIBILITY Individual Students INTERVENTIONS (TIER 2/3) Groups of Students Individual Students We ask the same questions at all levels 24 CORE (TIER 1) District Level School Level Grade Level

The Problem Solving Process How is it working? 4. Plan Implementation & Evaluation What are we going to do about the problem? 25 1. Problem Identification Improved Student Achievement 3. Plan Development What is the problem? 2. Problem Analysis Why is the problem occurring?

1. Problem Identification There is a difference between where we are and where we want to be Where we are Where we want to

District/School Where we want to be Grade Level Intervention Group Individual Student Where we are

1. Problem Identification Do your teachers “trust the data? “I think…” “I feel like…” “I believe…” VS. “Our data/eviden ce indicates… ” How clearly are you currently defining “problems” in your district? Do you primarily rely on the data?

2. Problem Analysis • Hypotheses development – Analyze WHY the problem might be happening/What is causing the problem…. Plan Development Problem Identification Problem Analysis

Focus on “the water” • Instruction • Curriculum • Environment 30 E C I

Student Learning Instruction: How you teach Environment: Where you teach 31 Curriculum: What you teach Learner: Who you teach

When it comes to problem analysis, just remember… ICE, ICE baby I – Instruction C – Curriculum E – Environment then L – Learner 32

Which ones make sense? Which ones are What impacts student achievement? surprising? Effective teaching variables Effect size Other variables Effect size Formative Evaluation +0. 90 Socioeconomic Status +0. 52 Comprehensive interventions for students with LD +0. 49 +0. 77 Parental Involvement Computer Based Instruction +0. 37 Teacher Clarity +0. 75 Feedback +0. 75 School Finances +0. 23 +0. 72 Class Size +0. 21 +0. 61 Whole Language +0. 06 Teacher-Student Relationships Problem Solving Teaching Direct Instruction +0. 59 Retention -0. 16 John Hattie, Visible Learning, 2016

2. Why is the problem happening? District/Sch ool Grade 2 nd graders Adopted weak in Curriculum decoding weak in (Curriculum ) & need phonics and more other foundational opportunities to respond skills (Instruction) Instruction? Intervention Group Students disengaged and off task Not getting enough feedback and OTR’s (Instruction ) & off task (Environmen t) & need more explicit Instruction Curriculum? (Environmen t) Environment?

Talk Time How much do your discussions about students focus on: • Instruction? • Curriculum? • Environment? • Learner?

3. Plan Development What action steps are needed to address the problem?

3. What can we do about the problem? District/S chool Proble Curriculu m m weak in Analysi phonics s Grade Interventio n Group Student Insufficient Students Weak disengaged & feedback & decoding OTR’s off task Curriculum (Environmen Instruction Insufficient t) Off task OTR’s Instruction Environmen Instruction not explicit t Purchase More focus Add Increase new on reinforcement feedback & curriculum decoding, system & OTR’s; add provide increase behavior Plan strong in phonics more OTR’s explicitness management

4. Plan Implementation & Evaluation Implement the plan & evaluate frequently

District/School Where we want to be Grade Level Intervention Group Individual Student Where we are

Problem Solving Process at all levels Tier 3: Individual Problem Solving Meetings FEW STUDENTS Tier 2/3: Intervention Review Meetings SOME STUDENTS Tier 1: Core Review Meetings ALL STUDENTS 41 What is the problem? How is it working? We ask the same questions at all levels Why is the problem occurring? What are we going to do about the problem?

Problem Solving is only successful when you have the right mindset

How will you train your staff? • Teachers are becoming better consumers of data, but still have difficulty with: – Determining which data sources are reliable/valid – Interpreting reliable/valid data and linking it to instruction – Knowing what to do with conflicting data • Teacher training on using data lags behind teacher training on instruction and classroom management

Train your teams on data evaluation and problem solving Training is needed on how to read data, how to use decision rules, and how to have efficient meetings • Does your staff understand why we use data? • Does your staff understand why certain data sources are more useful than others? • Does your staff understand why we focus on the instruction, curriculum, and environment?