Welcome to the Workshop Note There is more

Welcome to the Workshop! Note: There is more information about what was covered in the workshop in the accompanying handouts. 1

What Do the Numbers Mean? Making Sense out of Outcome Data Donna Spiker Kathy Hebbeler SRI International Outcomes Workshop Albuquerque, NM April 2006

Transforming the Assessment Information into Data on Child Outcomes

Need to transform the data n n n Under any assessment option, state needs a way to transform the assessment information into the indicators/evidence statements. No assessment provides information directly on the 3 functional outcomes areas No assessment provides information directly on the 3 categories (a, b, c) in the OSEP indicators 4

Need to transform the data n n …if assessment process involves multiple sources of information …if more than one assessment is being used in the state 5

Input Time 1 Scores Time 2 Scores Cognitive Communication Social Adaptive Motor ≠ Desired Output OSEP Indicators Social % a, b, c Knowledge &… % a, b, c Meet Needs (for 100 s of children!) % a, b, c 6

Decisions related to transforming the assessment data n n n What is the process by which the data gets transformed? What is the rubric used to “roll up the data? Where does the transformation occur? q State q Local program 7

A way to think about how children are doing with regard to each outcome Movement away from age -expected Age-expected skills and behavior Movement toward ageexpected 8

9

Definitions for Outcome Ratings 10

11

Topic 1 Using Assessment Data to Contribute Information to the ECO Summary Rating 12

Table: Item Results for 5 Children Items Related to Outcome 1 1 Plays well with others 2 Cooperates with peers in simple games 3 Stops for transition cues 4 Takes directions well from adults 5 Has at least one close friend Carlos A E E A A Geeta NY NY NY Eileen A A A Ming E E E NY NY Shaniqua E E A Name 13

Activity 1. Assign a Summary Rating to each child based on the score pattern. Table: Item Results for 5 Children Items Related to Outcome 1 Child Name 1 2 3 4 5 Carlos A E E A A Geeta NY NY NY Eileen A A A Ming E E E NY NY Shaniqua E E A Summary Rating 14

Topic 1. 1 Adding Value for Local Programs: Reports Produced for Local Programs (Note: This applies if assessment data are entered into an online system or if program will be producing reports in other ways) 15

Table: Outcome Ratings for Class 3 c by Child Name Carlos Geeta Eileen Ming Shaniqua Outcome 1 Outcome 2 Outcome 3 Time 1 Time 2 5 6 3 4 5 5 1 2 2 2 3 4 7 7 7 2 2 4 4 2 3 3 4 5 5 16

Table: Percentage of Children Scoring 5 or Higher by Class (Example of an Aggregated Report for Program) Outcome 1 Outcome 2 Outcome 3 Class Time 1 Time 2 1 a 65 70 50 51 49 52 1 b 55 53 62 61 87 88 2 a 47 43 51 67 65 66 2 b 76 84 78 85 78 83 3 a 97 98 95 97 95 100 17

Topic 2 Basic State Reports: Using Data from the Summary Rating to Produce OSEP Data 18

OSEP Indicators a. % of children who reach or maintain functioning at a level comparable to same-age peers More Powerful Alternative a 1. % of children who made sufficient progress to maintain functioning at a level comparable to same age peers a 2. % of children who made sufficient progress to achieve functioning at a level comparable to same age peers b. % of children who improve functioning but are not in a b 1. % of children who mover nearer to functioning comparable to same age peers but did not achieve it b 2. % of children who made progress but not sufficient progress to move nearer to functioning comparable to same-age peers c. % of children who did not improve functioning 19

Activity 2. Be a computer. Assign an ECO Category and an OSEP Category to the records below. State data file ID 343421 343422 343423 343425 343432 343433 343446 343450 343456 343459 Program Date of Entry Rating Q 1 a Date of Exit Rating Q 1 a Q 1 b 10/20/06 7 6/7/06 7 yes 11/1/06 6 8/9/06 7 yes 12/3/06 3 7/8/06 5 yes 12/3/06 4 10/26/07 4 yes 12/5/06 1 12/5/07 1 no 12/15/06 2 9/15/07 4 yes 1/5/07 1 8/6/07 1 yes 1/8/07 3 12/15/07 5 yes 1/25/07 4 10/12/07 7 yes 1/28/07 3 11/14/07 5 yes 11/22/06 4 12/20/07 3 yes ECO Category OSEP Category 71 71 71 20

FMA Findings Meaning Action 21

Table: Outcome 1: Percentage of Children in Each OSEP Category, 2008 -2010 OSEP Category 2008 N= 1345 2009 N=1409 2010 N=1480 a 1 15 23 22 a 2 30 25 22 b 53 52 55 c 2 2 1 22

Table: Outcome 1: Percentage of Children in Each OSEP Category by Local District A District B District C District D District E District F District G Category Statewide a 1 10 5 12 3 8 a 2 35 35 30 40 43 35 b 53 56 56 49 45 47 52 56 c 2 4 3 1 10 1 2 1 OSEP 23

Topic 3 Using Data from the Summary Rating to Produce Reports for the State – No Additional Data 24

Table: Percentage of Children in Different Progress Categories 2007 N=1345 2008 N=1409 2009 N=1480 Maintained age-appropriate 18 17 15 Achieved age-appropriate 27 29 29 Moved closer to age-appropriate 45 46 50 Made progress 8 6 5 Did not make progress 2 2 1 Category 25

Table: Percentage of Children Who Maintained or Moved Closer to Age-Appropriate Development, 2007 -2009 2007 N=1345 2008 N=1409 2009 N=1480 90 92 94 26

Table: Comparison of Populations at Entry and Exit Outcome 1 Level Entry (%) Exit (%) 7 (age-appropriate) 70 78 6 (age-appropriate) 22 15 5 (not age-appropriate) 4 4 4 (not age-appropriate) 1. 6 1. 2 3 (not age-appropriate) 1. 2 . 8 2 (not age-appropriate) . 8 1 (not age-appropriate) . 4 . 2 6. 54 6. 65 Mean/Average N= 500 27

Table: Progress of Children Between Fall and Spring Outcome 1 Progress N % Maintained age-expected functioning 350 70 Maintained same level function, but not age-expected 60 12 Gained 3 steps 10 2 Gained 2 steps 25 5 Gained 1 step 50 10 Dropped 1 step 4 . 8 Dropped 2 steps 1 . 2 28

Table: Using the Summary Rating Outcome 1 Level N % 7 (age-appropriate) 350 70 6 (age-appropriate) 110 22 5 (not age-appropriate) 20 4 4 (not age-appropriate) 8 1. 6 3 (not age-appropriate) 6 1. 2 2 (not age-appropriate) 4 . 8 1 (not age-appropriate) 2 . 4 29

Table: Outcome 1 at entry Item Number Item % Achieved % Emerging % Not Yet 1 Plays well with others 72 22 6 2 Cooperates with peers in simple games 67 30 3 3 Stops for transition cues 55 37 8 4 Takes directions well from adults 81 14 5 5 Has at least one close friend 85 8 7 30

Topic 4 Using Outcomes Data to Produce Reports for the State – Additional Data Available 31

Examples of Additional Variables n n n n child’s age at assessment length of time between assessment completions child’s condition, delay, or eligibility criterion and/or severity (if applicable) child’s teacher or care coordinator type and/or intensity of services received length of time in program quality rating on program 32

Two ways to get additional variables 1. merging the outcomes data with other data or 2. collecting additional data as part of the outcomes data collection 33

Table: Extent Of Change for Outcome 1 by District For Children Who Entered Programs Between 16 and 20 Months and Exited between 30 and 36 Months Children - Entry 16 -20 Months and Exit 30 -36 Months District 100 % of children District 101 % of children Maintained ageexpected functioning 70 63 Maintained same level function, but not ageexpected 12 15 Gained 3 steps 2 1 Gained 2 steps 5 3 Gained 1 step 10 17 Dropped 1 step . 8 1 Dropped 2 steps . 2 0 34

Table: Outcome 1 Rating at Exit by Age at Entry into the Early Intervention System Age at Entry into Intervention System Birth-6 Months % of children 9 -12 Months % of children 7 12 21 31 28 6 37 39 37 42 5 9 12 8 11 4 11 10 11 9 3 14 9 6 6 2 10 6 5 3 1 7 3 2 1 Outcome Score at 3 Yr. Exit 16 -20 Months 24 -30 Months % of children 35

Table: Outcome 2 Rating by Average Amount of Speech Therapy Received per Week Outcome Score at 3 Yr. Exit Average Amount of Speech Therapy Received none % of children Less than 30 minutes % of children 31 – 60 minutes % of children More than 60 minutes % of children 7 12 21 31 28 6 37 39 37 42 5 9 12 8 11 4 11 10 11 9 3 14 9 6 6 2 10 6 5 3 1 7 3 2 1 36

Summary and Take Away Messages n The value of the outcome data depends on what a state does with it. n n n Minimal value = meeting the federal reporting requirement Maximal value = meeting the needs of a variety of audiences in the state Outcomes data has the potential to answer many important questions. 37

Summary and Take Away Messages (Continued) n There are many different audiences for outcomes data. n n n Federal government State administrators State policy-makers Local administrators Providers and teachers Families 38

Summary and Take Away Messages (Continued) n n Different audiences have different questions and need the data reported in different ways. Thoughtful planning for the kinds of reports the state wants for different audiences is necessary to insure the system is built to meet the needs of different audiences. 39

Summary and Take Away Messages (Continued) n n The major cost in building an outcomes system is in the data collection. Generating one or 101 reports adds minimally to the cost. States need to maximize their investments by making sure the system is going to produce the answers (the reports) they want. 40

Just Remember…… If you have been good, the Data Fairy will bring reports to you!! For more information: www. the-eco-center-org. 41