Data Informed Decision Makers How to Use Data

Data Informed Decision Makers: How to Use Data For Decision Making Abby Schachner Megan Vinh Megan Cox Division for Early Childhood Annual Conference October 2017

Welcome! • Who is here? • What are your expectations for this session? 2

Agenda • National context • Creating a culture of data-informed decision-making – Including practitioner-driven data questions • How to efficiently and intentionally use data 3

National Context Results Driven Accountability • For over 30 years, there has been a strong focus on regulatory compliance based on the IDEA and Federal regulations for early intervention and special education – OSEP – States – Districts/Programs • As a result, compliance has improved! 4

Compliance improves, but not results? • Despite this focus on compliance, states are not seeing improved results for children and youth with disabilities: – Young children are not coming to Kindergarten prepared to learn – In many locations, a significant achievement gap exists between students with disabilities and their general education peers – Students are dropping out of school – Many students who do graduate with a regular education diploma are not college and career ready

National Context • Three joint policy statements have been released: – Policy Statement on Inclusion of Children with Disabilities in Early Childhood Programs – Policy Statement on Expulsion and Suspension Policies in Early Childhood Settings – Policy Statement on Family Engagement • Joint Policy Statements – – http: //www. acf. hhs. gov/programs/ecd/child-health-development/reducing-suspension-and-expulsionpractices http: //www 2. ed. gov/policy/speced/guid/earlylearning/joint-statement-full-text. pdf

What do these all have in common? • They all require the use of data to inform planning and to systemically improve results for children with disabilities and their families.

CREATING A CULTURE OF DATA -INFORMED DECISION MAKING

Practices • DEC Recommended Practices – Leadership promotes data-informed decisionmaking by creating a culture of evidence-centered policies and professional development opportunities that promote the implementation of the recommended practices.

Data – It’s a Leadership Team Responsibility • Monthly review of data – Who, How often, What, Where, When • Monthly review of program incidents – What’s up, what’s down, why, what should we do about it • Review of all teacher fidelity measures to determine next steps, training, coaching, support • Review of child progress data to ensure supports are effective

Cultural Challenges to Data-Informed Decision Making • Many providers/teachers have developed their own personal metric for judging the effectiveness of their intervention/teaching and often this metric differs from the metrics of external parties (e. g. , state accountability systems and school boards). • Many providers/teachers and administrators base their decisions on experience, intuition, and anecdotal information (professional judgment) rather than on information that is collected systematically. • There is little agreement among stakeholders about what kinds of data are meaningful and what to prioritize. • Some providers/teachers disassociate their own performance and that of children, which leads them to overlook useful data. Ingram, D. S. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106(6), 1258– 1287.

Technical Challenges Data-Informed Decision Making • Data that providers /teachers want – about outcomes, services and quality – are rarely available and are usually difficult to measure. • Programs and schools rarely provide the time needed to collect and analyze data. • Providers/teachers and/or administrators lack the access or capacity to analyze data for program improvement. Ingram, D. S. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106(6), 1258– 1287.

Political Challenges Data-Informed Decision Making • Data have often been used politically, leading to mistrust of data and data avoidance. • Providers/teachers and administrators may worry about the way data will be used to penalize them.

Discussion Questions. Small Group Activity • What are your barriers to creating a culture of datainformed decision making? – What are your potential solutions? • What do you struggle with in being a data-informed decision-maker? – What are your potential solutions?

EFFICIENTLY AND INTENTIONALLY USING DATA This Photo by Unknown Author is licensed under CC BY-SA

Key Concepts for Data-Informed Decision-Making • What are your questions? • What is your process for looking at data and making interpretations? • What are the data sources you might have? Is there other data you need to collect or gather?

Starting with a question (or two…) • All analyses are driven by questions • Questions come from different sources • Different versions of the same question are necessary and appropriate for different audiences. What are your critical questions? What questions might practitioners have?

Defining Data Analysis Questions What are your crucial policy and programmatic questions? Example: 1. Does our program remove some children more often than others? a. Are children with different racial/ethnic backgrounds removed at similar rates?

What is Your Process for Looking at Data? Evidence Inference Action 19

Evidence • Evidence refers to the numbers, such as “ 35% of boys have been removed at least once” • The numbers are not debatable 20

Inference • How do you interpret the evidence? • What can you conclude from the numbers? • Does evidence mean good news? Bad news? News you can’t interpret? • To reach an inference, sometimes you need to analyze data in other ways (ask for more evidence) 21

Inference • Inference is debatable -- even reasonable people can reach different conclusions • Stakeholders and having a variety of perspectives can help with putting meaning on the numbers • Early on, the inference may be more a question of the quality of the data 22

Action • Given the inference from the numbers, what should be done? • Recommendations or action steps • Action can be debatable – and often is • Another role for stakeholders and teams • May involve looking at additional data and information • Again, early on the action might have to do with improving the quality of the data 23

Preparation – Plan to Succeed • • • Define purpose and the issue Identify who needs to be involved Timelines Identify relevant questions Identify relevant data Generate hypotheses Evidence – Dig Into Data • Analyze the data • Develop methods and materials for displaying the data Action – Contribute to Success Inference – Interpret & Share With Others • Celebrate success • Develop & implement improvement plans • Evaluate progress • Share data materials • Check support for hypotheses • Connect inferences with root causes

WHAT ARE YOUR DATA SOURCES? Data: facts or information used usually to calculate, analyze, or plan something

Small Group Activity • Articulate your question • Evidence: Critically examine the data provided • Inference: Discuss what inferences you can make • Action: Brainstorm potential actions and next steps

Wrap-Up • What were your inferences and actions based on the data? • Did you have any ah-has? • Reflections on the process and experience?

Intentionality Leads to Success • Think about how you can maximize data you already collect and collect what you need • Think about how to organize your staff and your agency around ongoing data use • Its all about continuous improvement • Use data to determine priority for focus • It is important to “drill down” to understand performance to identify meaningful solutions

Contact • Abby Schachner, abby. schachner@sri. com • Megan Vinh, mvinh@email. unc. edu • Megan Cox, megan. cox@sri. com • Da. Sy Center website: http: //dasycenter. org/ • ECTA Center website: http: //ectacenter. org/

Resources • Da. Sy Critical Questions for Early Intervention and Early Childhood Special Education – http: //dasycenter. org/critical-questions-about-early-intervention-and-early-childhood-specialeducation/ • Planning, Conducting, and Documenting Data Analysis for Program Improvement – http: //dasycenter. org/planning-conducting-and-documenting-data-analysis-for-programimprovement/ • Head Start Modules on Creating a Culture that Embraces Data – http: //eclkc. ohs. acf. hhs. gov/hslc/tta-system/operations/data/guide. html • Data Visualization Toolkit – http: //dasycenter. org/data-visualization-toolkit/ • Prevent Expulsion – http: //preventexpulsion. org/ • RP 2 materials – http: //ectacenter. org/implement_ebp. asp • Inclusion self-assessment – http: //ectacenter. org/~pdfs/topics/inclusion/ecta-dasy_inclusion-online-self-assessment_05 -17 -17. pdf

The contents of this presentation were developed under a grant from the U. S. Department of Education, # H 373 Z 120002, and a cooperative agreement, #H 326 P 120002, from the Office of Special Education Programs, U. S. Department of Education. However, those contents do not necessarily represent the policy of the U. S. Department of Education, and you should not assume endorsement by the Federal Government. Da. Sy Center Project Officers, Meredith Miceli and Richelle Davis and ECTA Center Project Officer, Julia Martin Eile.