Getting Started ValueAdded Analysis Copyright 2006 Battelle for
Getting Started Value-Added Analysis Copyright © 2006. Battelle for Kids.
Presentation Targets • Understand the difference between progress and achievement. • Gain an introductory understanding of value-added analysis. • Acquire an awareness of information available in value-added reports. • Explore how value-added information can inform school improvement. Copyright © 2006. Battelle for Kids.
Stair-Step Expectations Grade Three Grade Four Grade 3 Test Grade 4 Test Grade Five Grade 5 Test In a perfect world scores on year-end achievement tests would be enough to document growth. Students would start school at the same place, and move together toward proficiency. Copyright © 2006. Battelle for Kids. Copyright © 2003. Battelle for Kids
Differentiated Reality Grade Three Grade Four Grade 3 Test Grade 4 Test Grade Five Grade 5 Test Students begin school at different places and progress at different rates. This complicates accountability. We need more than scores on a single test to tell the story of a school’s effectiveness. Copyright © 2006. Battelle for Kids. Copyright © 2003. Battelle for Kids
A View of Student Progress: Accountability perspective Stude nt A Proficient nt Stude Copyright © 2006. Battelle for Kids. B We must expect progress for all students.
The question for educators today Regardless of where students begin, how do we maximize the progress each makes in any given year? 84% of the American public believe student progress is the best measure of a school’s effectiveness! Copyright © 2006. Battelle for Kids. Copyright © 2003. Battelle for Kids
What is the difference between progress and achievement? Copyright © 2006. Battelle for Kids.
Value-Added Analysis in Ohio Project SOAR Ohio’s Value-Added System Copyright © 2006. Battelle for Kids.
Ohio’s System • Operated by the Ohio Department of Education. • Began as a grade 4 pilot in 2006 in all districts and community schools. • Provides analysis in math & reading, grades 4 -8. • Uses only state achievement test data. • Uses the mean gain model. • Expected growth is set by the State Board of Education. Copyright © 2006. Battelle for Kids.
Project SOAR (Schools’ Online Assessment Reports) • Operated by Battelle for Kids. • Began in 2002 with 42 school districts. • Now includes 100+ districts. • Provides analysis in all core subjects, grades 3 -10. • Uses state and non-state test data. • Uses the mean predicted model. • Expected growth is normative (Average Growth). Copyright © 2006. Battelle for Kids.
Value-Added in the State Accountability System • If a district or school demonstrates more than expected growth over two consecutive years it moves up one level in the system. • If a district or school demonstrates less than expected growth over three consecutive years it moves down one level in the system. • Value-added measures are significant at one standard error. Copyright © 2006. Battelle for Kids.
How is value-added used in the state accountability system? Copyright © 2006. Battelle for Kids.
Value-Added Analysis* Conceptual Overview How is student assessment data used to produce a value-added growth metric? Copyright © 2006. Battelle for Kids.
What is a Growth Measure? Growth = Observed Score – Baseline Score Copyright © 2006. Battelle for Kids.
Value-Added Example 4 th 5 th 6 th 7 th 8 th A value-added metric measures the average growth that is produced in one subject in a given year. This growth is then compared to the amount of growth that would be expected for that group of students. Growth above (or below) expectation is the “value-added. ” Copyright © 2006. Battelle for Kids.
Test Properties Tests must: • Be highly correlated to curricular objectives. • Have sufficient stretch. • Have appropriate levels of reliability. Copyright © 2006. Battelle for Kids.
Mean Gain Approach Used when all tests across grade levels have consistent scales. School Effect ≈ Mean Gain – Expected Growth Copyright © 2006. Battelle for Kids.
Mean Gain Example Math scaled scores for the same students. Year 1 • Grade 3 Student 1 377 = 26 (NCE) Student 2 380 = 27 Student 3 360 = 19 Student 4 377 = 26 Student 5 380 = 28 Year 2 • Student 1 Student 2 Student 3 Student 4 Student 5 Grade 4 400 = 37 (NCE) 385 = 30 395 = 35 405 = 41 390 = 33 Mean Baseline = 25. 2 Mean Observed = 35. 2 Growth = Mean Observed – Mean Baseline Growth = 35. 2 - 25. 2 = 10. 0 A crude measure of the growth for this group is 10 NCEs. Copyright © 2006. Battelle for Kids.
Mean Gain Approach (Converted to NCEs) Sample School Value-Added Report Math High Progress Positive School Effects High Achievement High Mean NCE or Mean Scaled Score School Effect Baseline Observed Copyright © 2006. Battelle for Kids.
Describe, in conceptual terms, the “mean gain” value-added model. Copyright © 2006. Battelle for Kids.
What New Diagnostic Information Do Value-Added Reports Provide?
Value-Added Summary Report Where are the areas of strength and weakness? Copyright © 2006. Battelle for Kids.
School Diagnostic Report Are the curriculum and instruction working for all students? Copyright © 2006. Battelle for Kids.
Performance Diagnostic Report What pattern would represent optimum growth? Copyright © 2006. Battelle for Kids.
Student Report How does a student’s growth rate compare to other groups of students? Copyright © 2006. Battelle for Kids.
Student Search How can Predictions help identify at risk students? Copyright © 2006. Battelle for Kids.
Student Search Results How could this information change the way educators approach intervention? Copyright © 2006. Battelle for Kids.
In Summary, Value-Added Information Shows… • The students’ progress made in each subject area and grade level. • The progress students made who enter the year at different achievement levels. • How students’ progress in one curricular area or program compares to their progress in another. • Whether individual students are making adequate progress toward state standards. Without data, all we have are opinions! Copyright © 2006. Battelle for Kids.
What new information do value-added reports provide? Copyright © 2006. Battelle for Kids.
Connections with School Improvement Efforts How can value-added information be used in an annual school improvement cycle? Copyright © 2006. Battelle for Kids.
Comparison of Effect Sizes Copyright © 2006. Battelle for Kids.
Beginning-of-the-Year Planning Grade Level/Department Teams • Examine value-added and other school performance information. • Assess strengths and weaknesses and their potential causes. • Celebrate strengths. • Set goal(s) that build on strengths and address weaknesses. • Create action plans, timelines and responsibilities. Copyright © 2006. Battelle for Kids.
Using Value-Added Information in an Annual School Improvement Cycle Beginning-of-the-Year Planning Teams meet to work on goals Individual teachers working on the work Teams meet to work on goals Copyright © 2006. Battelle for Kids. Teams meet to work on goals
Copyright © 2006. Battelle for Kids.
Why is it important to focus school improvement strategies at the individual grade level and/or subject area? Copyright © 2006. Battelle for Kids.
For more information, contact: (866) KIDS-555 www. Battellefor. Kids. org
- Slides: 36