Improving and Interpreting Child Outcomes Data Integrating Outcomes

Improving and Interpreting Child Outcomes Data Integrating Outcomes Learning Community Call February 8, 2012 1

Objectives for the call • To discuss strategies and resources for improving child outcomes data quality • To focus on looking at data through pattern checking as a mechanism for improving data quality Early Childhood Outcomes Center 2

Quality Assurance: Looking for Quality Data I know it is in here somewhere Early Childhood Outcomes Center 3

Ongoing checks for data quality Good training and assessment Before During After Early Childhood Outcomes Center Efficient data systems Timely and accurate data entry Ongoing supervision of implementation Feedback to implementers Refresher training Review sample of completed COSFs Pattern checking analysis 4

Quality checks before data collection begins Good training and assessment Before Efficient data systems Timely and accurate data entry Early Childhood Outcomes Center 5

Promoting Quality Data Through training and communication (before and during data collection) related to: – Assessment – Understanding the COSF process – Age expectations – Data entry Early Childhood Outcomes Center 6

Promoting Quality Data Through data systems and verification, such as – Data system error checks – Good data entry procedures Early Childhood Outcomes Center 7

Quality checks during data collection During Ongoing supervision of implementation Feedback to implementers Refresher training Early Childhood Outcomes Center 8

Ongoing supervision • Review of the process – Is the process high quality? – Are teams reaching the correct rating? • Methods – Observation – Videos e. g. Quality review of COS team discussion Early Childhood Outcomes Center 9

Ongoing Supervision • Feedback to teams is critical • Refresher training • Beware of – Auto pilot – Drift Early Childhood Outcomes Center 10

Quality review through process checks • Provider surveys – Self assessment of competence – Knowledge checks – Process descriptions (who participates? ) – Identification of barriers Early Childhood Outcomes Center 11

Quality checks after data collection After Early Childhood Outcomes Center Review sample of completed COSFs Pattern checking analysis 12

Quality Review of Completed COS Forms • • • Early Childhood Outcomes Center complete? adequate evidence? match the outcome area? based on functional behaviors? across settings and situations? ratings consistent with evidence? 13

Pattern Checking Analysis The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns. Pattern Checking Table http: //www. fpg. unc. edu/~eco/assets/pdfs/Patter n_Checking_Table. pdf 14

Ongoing checks for data quality - Poll Good training and assessment Before During After Early Childhood Outcomes Center Efficient data systems Timely and accurate data entry Ongoing supervision of implementation Feedback to implementers Refresher training Review sample of completed COSFs Pattern checking analysis 15

Take Home Message If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else. What do you if the data are not as good as they should be? Answer: Continue to improve data collection through ongoing quality assurance Early Childhood Outcomes Center 16

Looking at Data Early Childhood Outcomes Center 17

Using data for program improvement = EIA Evidence Inference Action Early Childhood Outcomes Center

Evidence • Evidence refers to the numbers, such as “ 45% of children in category b” • The numbers are not debatable Early Childhood Outcomes Center 19

Inference • How do you interpret the #s? • What can you conclude from the #s? • Does evidence mean good news? Bad news? News we can’t interpret? • To reach an inference, sometimes we analyze data in other ways (ask for more evidence) Early Childhood Outcomes Center 20

Inference • Inference is debatable -- even reasonable people can reach different conclusions • Stakeholders can help with putting meaning on the numbers • Early on, the inference may be more a question of the quality of the data Early Childhood Outcomes Center 22

Action • Given the inference from the numbers, what should be done? • Recommendations or action steps • Action can be debatable – and often is • Another role for stakeholders • Again, early on the action might have to do with improving the quality of the data Early Childhood Outcomes Center 22

Promoting quality data through data analysis 23

Checking to see if ratings accurately reflect child status: Pattern checking • We have expectations about how child outcomes data should look – Compared to what we expect – Compared to other data in the state – Compared to similar states/regions/school districts • When the data are different than expected ask follow up questions Early Childhood Outcomes Center 24

Questions to ask • Do the data make sense? – Am I surprised? Do I believe the data? Believe some of the data? All of the data? • If the data are reasonable (or when they become reasonable), what might they tell us? Early Childhood Outcomes Center 25

Let’s look at some data … Early Childhood Outcomes Center

Remember: Child Outcomes (see reference sheet) 1. Positive social-emotional skills (including social relationships); 2. Acquisition and use of knowledge and skills (including early language/communication [and early literacy]); and 3. Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center 27

Remember: COSF 7 -point scale (see reference sheet) • • 7 -Completely- Age appropriate functioning in all or almost all everyday situations; no concerns 6 - Age appropriate functioning, some significant concerns 5 -Somewhat- Age appropriate functioning some of the time and/or in some settings and situations 4 - Occasional age-appropriate functioning across settings and situations; more functioning is not age-appropriate than age appropriate. 3 -Nearly- Not yet age appropriate functioning; immediate foundational skills most or all of the time 2 - Occasional use of immediate foundational skills 1 -Not yet- Not yet age appropriate functioning or immediate foundational skills Early Childhood Outcomes Center 28

Remember: Reporting Categories (see reference sheet) Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to sameaged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers Early Childhood Outcomes Center 29

Remember: Summary Statements (see cheat sheet) 1. Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program. c + d___ a+b+c+d Early Childhood Outcomes Center 30

Remember: Summary Statements (see cheat sheet) 2. The percent of children who were functioning within age expectations in each Outcome by the time they exited the program. d + e__ a+b+c+d+e Early Childhood Outcomes Center 31

National Data Early Childhood Outcomes Center 32

National Data for Early Childhood Special Education, 2009 -2010 40% 35% 30% 25% 20% 15% 10% 5% 0% a b c d e Social relationships 1, 7% 11, 5% 28, 2% 34, 3% 24, 3% Knowledge and skills 1, 8% 13, 4% 32, 5% 34, 4% 17, 9% Action to meet needs 1, 6% 10, 8% 20, 9% 35, 6% 31, 0% Note: Based on 33 States with highest quality data 33

National Summary Statements Data Early Childhood Special Education (619), 2009 -2010 90, 0% 80, 0% 70, 0% 60, 0% 50, 0% 40, 0% 30, 0% 20, 0% 10, 0% SS 1: Greater than expected growth SS 2: Exited within age expectations Social Relationships 82, 6% 58, 7% Knowledge and Skills 81, 5% 52, 3% Action to meet needs 81, 9% 66, 7% Note: Based on 33 States with highest quality data Early Childhood Outcomes Center 34

National Data for Early Intervention (Part C), 2009 -2010 40% 35% 30% 25% 20% 15% 10% 5% 0% a b c d e Social relationships 1, 7% 18, 0% 18, 6% 29, 4% 32, 4% Knowledge and skills 1, 5% 20, 0% 24, 8% 36, 9% 16, 8% Action to meet needs 1, 5% 17, 5% 21, 4% 37, 1% 22, 6% Note: Based on 29 States with highest quality data Early Childhood Outcomes Center 35

National Summary Statements Data Early Intervention (Part C), 2009 -2010 100, 0% 90, 0% 80, 0% 70, 0% 60, 0% 50, 0% 40, 0% 30, 0% 20, 0% 10, 0% Social Relationships Knowledge and Skills Action to meet needs SS 1: Greater than expected growth 71, 0% 74, 2% 75, 6% SS 2: Exited within age expectations 61, 8% 53, 8% 59, 7% Note: Based on 29 States with highest quality data Early Childhood Outcomes Center 36

Sample State Data Early Childhood Outcomes Center 37

Key to Good Data Have a good outcome measurement Early Childhood Outcomes Center 38

Quality Checks • Missing Data • Pattern Checking Full ECO Pattern Checking Table: http: //www. fpg. unc. edu/~eco/assets/pdfs/Pattern_Checking_Table. pdf Early Childhood Outcomes Center 39

Missing Data - Overall • How many children should the state be reporting to OSEP in the SPP/APR table? – i. e. , how many children exited in the year, and stayed in the program 6 months? – Do you have a way to know? • What percentage of those children do you have in the table? • Are you missing data selectively? (by local program, by child or family characteristic? ) Early Childhood Outcomes Center 40

Pattern Checking • The quality of the child outcomes data are established by a series of analyses that demonstrate the data are showing predictable patterns: – Across Outcomes – Across time – Compared with disability information Early Childhood Outcomes Center 41

Information available for pattern checking 1. 2. 3. 4. OSEP Progress Categories Entry Data Exit Data Summary Statements Early Childhood Outcomes Center 42

Predicted Pattern #1 1 a. Children will differ from one another in their entry scores in reasonable ways (e. g. , fewer scores at the high and low ends of the distribution, more scores in the middle). . 1 b. Children will differ from one another in their exit scores in reasonable ways. 1 c. Children will differ from one another in their OSEP progress categories in reasonable ways. Early Childhood Outcomes Center 43

Rationale Evidence suggests EI and ECSE serve more mildly than severely impaired children (e. g. , few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range). Early Childhood Outcomes Center 44

Predicted Pattern #1 (cont’d) Analysis 1. Look at the distribution of rating/scores at entry and exit and the data reported to OSEP. 2. Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit. Question: Is the distribution sensible? What do you expect to see? Early Childhood Outcomes Center 45

Exit Data Early Childhood Outcomes Center 46

Poll • Do you pull child outcomes data by the 1 -7 ratings on the scale? • What have you learned? Early Childhood Outcomes Center 47

Hypothetical State Data: Outcome 1 Exit Data Early Childhood Outcomes Center 48

Hypothetical State Data: Outcome 2 Exit Data Early Childhood Outcomes Center 49

Hypothetical State Data: Outcome 3 Exit Data Early Childhood Outcomes Center 50

Poll • Do you look at children exiting the program at age expectations across all three outcomes (6 & 7)? • What have you learned? Early Childhood Outcomes Center 51

Question? What would you anticipate as the percentage of children who leave your program and no longer need special education services? Early Childhood Outcomes Center 52

‘At Age’ across all three outcomes: OSEP progress categories • Percent of children that scored a 6 or 7 on all three outcomes at entry – %xx (n = xx) • Percent of children that scored a 6 or 7 on all three outcomes at exit – %xx (n = xx) Early Childhood Outcomes Center 53

Hypothetical State Data: Across all 3 outcomes Early Childhood Outcomes Center 54

Poll • Do you pull child outcomes data by OSEP progress categories? • Are you comparing your data to national or other states’ data? • What have you learned? Early Childhood Outcomes Center 55

OSEP Categories Early Childhood Outcomes Center 56

Hypothetical State Data: Outcome 1 OSEP Categories Early Childhood Outcomes Center 57

Hypothetical State Data: Outcome 2 OSEP Categories Early Childhood Outcomes Center 58

Hypothetical State Data: Outcome 3 OSEP Categories Early Childhood Outcomes Center 59

Predicted Pattern #4 b 4 b. Children will not show huge changes between entry and exit. Analyses: Comparison of entry and exit scores (exit score minus entry score) Question: What do we expect to see? Early Childhood Outcomes Center 60

Poll • Are you pulling your data to look at the number of children who increase by 4 or more points on the 7 point scale? • What have you learned? Early Childhood Outcomes Center 61

Outcome 1: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 62

Outcome 2: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 63

Outcome 3: Children that increased by 4 or more points from entry to exit Early Childhood Outcomes Center 64

Wrap-up Early Childhood Outcomes Center 65

Drilling down: Looking at data by local program • All analyses that can be run with the state data can be run with the local data • The same patterns should hold and the same predictions apply. • Need to be careful about the size of N with small programs. Early Childhood Outcomes Center 66

Are your data high quality? 1. Are the missing data less than 5%? 2. Do your state’s data support the predicted patterns? • • If not, where are the problems? What do you know or can you find out about why they are occurring? Early Childhood Outcomes Center 67

Questions to ask • Do the data make sense? – Am I surprised? Do I believe the data? Believe some of the data? All of the data? • If the data are reasonable (or when they become reasonable), what might they tell us? Early Childhood Outcomes Center 68

Discussion Early Childhood Outcomes Center 69

Find more resources at: http: //www. the-eco-center-org Early Childhood Outcomes Center 70