Summary Statements The problem Progress data included 5

Summary Statements

The problem. . . • Progress data included – 5 progress categories – For each of 3 outcomes – Total of 15 numbers reported each year • Too many interrelated targets to make sense of • OSEP asked for a recommendation 2

Thinking through the summary statements • ECO presented options to states and ECO work groups via conference calls • Two sessions at December, 2008 EC Conference • Posted on the ECO web site for comments • ECO made recommendation to OSEP 3

Final Deliberation • OSEP put the summary statements out for public comment • Comments came in that were thoughtful, but not necessarily consistent with one another • Advantages and disadvantages to all options 4

• Paper documenting the process on the ECO website • Setting Targets for Child Outcomes 5

The Summary Statements • Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program. • The percent of children who were functioning within age expectations in each Outcome by the time they exited the program. 6

Example of State Progress Data for 2008 -2009 Number of children % of children a. Percent of infants and toddlers who did not improve functioning 40 4 b. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 150 15 c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach 270 27 d. Percent of infants and toddlers who improved functioning to reach a level comparable to sameaged peers 300 30 e. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers Total 240 24 N=1000 100% Positive social-emotional skills (including social relationships):

Summary Statement Data • Required Summary Statement 1: Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program= 75% • Required Summary Statement 2: The percent of children who were functioning within age expectations in each Outcome by the time they exited the program= 54% 8

Where do the #s come from? • Measurement for Summary Statement 1: Percent = # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in category (d) divided by [# of infants and toddlers reported in progress category (a) plus # of infants and toddlers reported in progress category (b) plus # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in progress category (d)] times 100. 9

Where do the #s come from? Prog cat # % a 40 150 270 300 240 4 15 27 30 24 b c d e 760 (a, b, c, and d) or 76% of the children entered the program functioning below age expectations 240 (e) or 24% of the children entered and exited functioning at age expectations 10

Where do the #s come from? Prog cat # % a b c 40 150 4 15 d e 270 27 300 30 24 570 (c and d) of the 760 (a, b, c, and d) changed their growth trajectories (made greater than expected progress) 270 +300= 570 = 75% 760 11

Where do the #s come from? Summary Statements Calculator -April 14, 2009 12

Where do the #s come from? • Measurement for Summary Statement 2: Percent = # of infants and toddlers reported in progress category (d) plus [# of infants and toddlers reported in progress category (e) divided by the total # of infants and toddlers reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100. 13

Where do the #s come from? Prog cat # % a b c d 40 150 270 4 15 27 e 300 30 24 30% of the children reached age expectations by exit and 24% of the children entered and exited at age expectations 300+240= 540 = 54% 1000 14

So remind me again what this means What can we say about the children’s progress? 15

What can we say? • Part C Outcome 1: successful social relationships with peers and adults, following rules for social interactions • 96% of children participating in Part C made progress in their social relationships while they were enrolled. • The 4% of children who did not make progress included children with the most severe disabilities and/or degenerative conditions. Can you describe 16 them?

• 24% of the children participating in Part C were functioning at age expectations at entry and at exit in this outcome area. Can you describe them? • 54% of the children were functioning at age expectations in this outcome area when they exited the program. (summary statement 2) – 30% started out behind and caught up – 24% entered and exited at age expectations 17

• 75% of the children who entered the program below age expectations made greater than expected gains, made substantial increases in their rates of growth. i. e. changed their growth trajectories (summary statement 1) 18

What other data might you want to share? • The public likes to see scores going up! • Increase in mean scores from entry to exit – e. g. 4. 3 to 5. 6 on the COSF – Increase in raw scores – Increase in scale scores • What else? 19

Setting Targets

What we’ll cover today • Two strategies for examining data – Data quality – Potential for program improvement • Parameters, guidance for target settings from OSEP 21

Can you trust the data? • Begin by identifying outliers • Examples: look at the percentages reported for certain categories across local programs 22

Percentages reported in category “a” across 30 local programs 30 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 23

Remove the outliers 30 %a 25 • State percentage for “a” with all data= 3. 9% 20 15 • Revised percentage for “a” with outliers removed= 2. 4% 10 5 0 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930 24

Percentages reported in category “e” across 30 local programs 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 25

Remove the outliers 70 %e 60 • State percentage for “e” with all data= 32. 1% 50 40 • Revised percentage for “e” with outliers removed= 27. 7% 30 20 10 0 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930 26

Example of data with outliers removed Progress Category Original % Clean % a 4 15 2 17 27 30 24 75 54 30 31 20 76 51 b c d e Sum St 1 Sum St 2 Clean data (without the outliers) may be a more accurate picture of where you are starting 27

Suggested strategy • Analyze your data with your local LEA/program outliers included and excluded so you can gauge the impact they are having on your state level data.

Note • Consider clean data when deciding about reasonable targets, BUT • Turn in the original data to OSEP in the SPP report! • You can discuss the clean data in the rationale for your targets. 29

Which local programs can be targeted for program improvement? • Compare the summary statement data by local program to identify which programs have the most potential for improvement. 30

Summary Statement Percentages by Local Program 100 90 80 70 60 50 Sum St 1 40 Sum St 2 30 20 10 0 prog c prog s prog j prog t prog a prog q prog p prog f prog h prog l prog b prog g prog r prog e prog k prog i prog m prog o prog w prog u prog d prog v prog n 31

Considerations • What do you know about the programs/LEAs with the least and the most progress in the summary statements? i. e. the programs w/ – the lowest and highest percentages of children at age expectation at exit – the lowest and highest percentage of children making greater than expected gains 32

Examples of Key Questions • Are the children similar at entry? • Are the higher performing programs/LEAs participating in special projects? e. g. a state initiative, TACSEI or CELL? • Are there systems issues in lower performing programs/LEAs that would explain differences in outcomes? e. g. personnel shortages 33

Bottom-line Question • Could either system or practice focused improvement activities targeted toward the lowest performing programs/LEAs improve the child outcomes? 34

The Math of Target Setting • How much would the data change if the lowest local programs moved toward the mean? • Improvements in the lowest programs will result in improvement in your statewide data • Experiment with your data to determine what targets are reasonable in your state 35