WISCONSIN STATEWIDE VALUEADDED TRAINING CESA 1 ValueAdded Research

WISCONSIN STATEWIDE VALUE-ADDED TRAINING CESA #1 Value-Added Research Center (VARC) Leslie Steinhaus, Sean Mc. Laughlin, Ernest Morgan March 2012

Agenda for Monday Several of these sections are new content for VARC’s presentation s. Please fill out the feedback forms. Fill it out as we go, or wait until the end. Value-Added Context Value-Added Basics – The Oak Tree Analogy Oak Tree Additions Education Based Examples Value-Added Report Interpretation and Decision Making Quadrant Exercise Data - Your Reports

Feedback Form Thank you for helping us improve our sessions.

Value-Added Context Value-Added Research Center Projects The Goal of Value-Added in Education Using Value-Added with Other Data Sean Mc. Laughlin

Districts and States Working with VARC NORTH DAKOTA MINNESOT A Minneapolis SOUTH DAKOTA WISCONSI N Milwaukee Madison Racine ILLINOIS Denver Chicago New York City Tulsa Los Angeles Atlanta Hillsborough County Collier County

Value-Added in Wisconsin

Achievement and Value-Added For the most complete picture of student and school performance, it is best to look at both Achievement and Value-Added. This will tell you � What students know at a point in time (Achievement) � How your school is affecting student academic growth (Value-Added)

The Power of Two Achievement Compares students’ performance to a standard Does not factor in students’ background characteristics Measures students’ performance at a single point in time Critical to students’ postsecondary opportunities & A more complete picture of student learning Value-Added Measures students’ individual academic growth longitudinally Factors in students’ background characteristics outside of the school’s control Measures the impact of teachers and schools on academic growth Critical to ensuring students’ future academic success Adapted from materials created by Battelle for Kids

Value-Added Model Description Design Output Objective • Quasi-experimental statistical model • Controls for nonschool factors (prior achievement, student and family characteristics) • Productivity estimates for contribution of educational units (schools, classrooms, teachers) to student achievement growth • Valid and fair comparisons of school productivity, given that schools may serve very different student populations

Value-Added Basics – The Oak Tree Analogy Sean Mc. Laughlin

The Oak Tree Analogy

Explaining the concept of value added by evaluating the performance of two gardeners • For the past year, these gardeners have been tending to their oak trees trying to maximize the height of the trees. • Each gardener used a variety of strategies to help their own tree grow. We want to evaluate which of these two gardeners was more successful with their strategies. Gardener B Gardener A Gardener B

To measure the performance of the gardeners, we will measure the height of the trees today (1 year after they began tending to the trees). • Using this method, Gardener B is the better gardener. This method is analogous to using an Achievement Model. 72 in. Gardener B Gardener A 61 in.

Pause and Reflect How is this similar to how schools have been judged in Wisconsin? What information is missing?

… but this achievement result does not tell the whole story. • These trees are 4 years old. • We need to find the starting height for each tree in order to more fairly evaluate each gardener’s performance during the past year. • The trees were much shorter last year. 72 in. Gardener B Gardener A 61 in. 47 in. Oak A Age 3 (1 year ago) Oak A Age 4 (Today) 52 in. Oak B Age 3 (1 year ago) Oak B Age 4 (Today)

We can compare the height of the trees one year ago to the height today. • By finding the difference between these heights, we can determine how many inches the trees grew during the year of gardener’s care. • Oak B had more growth this year, so Gardener B is the better gardener. This is analogous to a Simple Growth Model, also called Gain. Gardener A 47 in. Oak A Age 3 (1 year ago) n. 61 in. i 4 +1 Oak A Age 4 (Today) . 52 in. Oak B Age 3 (1 year ago) +2 n 0 i 72 in. Gardener B Oak B Age 4 (Today)

… but this simple growth result does not tell the whole story either. • We do not yet know how much of this growth was due to the strategies used by the gardeners themselves. • This is an “apples to oranges” comparison. • For our oak tree example, three environmental factors we will examine are: Rainfall, Soil Richness, and Temperature. Gardener A Gardener B

External condition Oak Tree A Oak Tree B Rainfall amount High Low Soil richness Temperature Gardener A Gardener B

We can use this information to calculate a predicted height for each tree today if it was being cared for by an average gardener in the area… • We examine all oaks in the region to find an average height improvement for trees. • We adjust this prediction for the effect of each tree’s environmental conditions. • We compare the actual height of the trees to their predicted heights to determine if the gardener’s effect was above or below average. Gardener A Gardener B

In order to find the impact of rainfall, soil richness, and temperature, we will plot the growth of each individual oak in the region compared to its environmental conditions.

Now that we have identified growth trends for each of these environmental factors, we need to convert them into a form usable for our predictions. Rainfall Low Medium High Growth in inches relative to the average -5 -2 +3 Soil Richness Low Medium High Growth in inches relative to the average -3 -1 +2 Temperature Low Medium High Growth in inches relative to the average +5 -3 -8 Now we can go back to Oak A and Oak B to adjust for their growing conditions.

To make our initial prediction, we use the average height improvement for all trees • Based on our data, the average improvement for oak trees in the region was 20 inches during the past year. • We start with the trees’ height at age 3 and add 20 inches for our initial prediction. • Next, we will refine our prediction based on the growing conditions for each tree. When we are done, we will have an “apples to apples” comparison of the gardeners’ effect. Gardener A 72 in. Gardener B 67 in. 52 in. 47 in. +20 Average Oak A Age 3 (1 year ago) Oak A Prediction Oak B Age 3 (1 year ago) Oak B Prediction

Based on data for all oak trees in the region, we found that high rainfall resulted in 3 inches of extra growth on average. For having high rainfall, Oak A’s prediction is adjusted by +3 to compensate. Similarly, for having low rainfall, Oak B’s prediction is adjusted by -5 to compensate. Gardener A 67 in. Gardener B 70 in. 47 in. 52 in. +20 Average + 3 for Rainfall - 5 for Rainfall

For having poor soil, Oak A’s prediction is adjusted by -3. For having rich soil, Oak B’s prediction is adjusted by +2. Gardener A 69 in. Gardener B 67 in. 47 in. 52 in. +20 Average + 3 for Rainfall - 5 for Rainfall - 3 for Soil + 2 for Soil

For having high temperature, Oak A’s prediction is adjusted by -8. For having low temperature, Oak B’s prediction is adjusted by +5. 74 in. Gardener A 59 in. 47 in. Gardener B 52 in. +20 Average + 3 for Rainfall - 5 for Rainfall - 3 for Soil + 2 for Soil - 8 for Temp + 5 for Temp

Now that we have refined our predictions based on the effect of environmental conditions, our gardeners are on a level playing field. The predicted height for trees in Oak A’s conditions is 59 inches. The predicted height for trees in Oak B’s conditions is 74 inches. 74 in. Gardener A 59 in. 47 in. Gardener B 52 in. +20 Average + 3 for Rainfall - 5 for Rainfall - 3 for Soil + 2 for Soil - 8 for Temp _____ +12 inches During the year + 5 for Temp _____ +22 inches During the year

Finally, we compare the actual height of the trees to our predictions. Oak A’s actual height of 61 inches is 2 inches more than we predicted. We attribute this above-average result to the effect of Gardener A. Oak B’s actual height of 72 inches is 2 inches less than we predicted. We attribute this below-average result to the effect of Gardener B. Gardener A +2 59 in. Predicted Oak A Actual Oak A 74 in. -2 72 in. Gardener B 61 in. Predicted Oak B Actual Oak B

Using this method, Gardener A is the superior gardener. By accounting for last year’s height and environmental conditions of the trees during this year, we found the “value” each gardener “added” to the growth of the tree. This is analogous to a Value-Added measure. Gardener A +2 59 in. 74 in. -2 72 in. Gardener B 61 in. Above Average Value-Added Predicted Oak A Below Average Value-Added Actual Oak A Predicted Oak B Actual Oak B

How does this analogy relate to value added in the education context? Oak Tree Analogy Value-Added in Education What are we evaluating? • Gardeners • Districts • Schools • Grades • Classrooms • Programs and Interventions What are we using to measure success? • Relative height improvement in inches • Relative improvement on standardized test scores Sample • Single oak tree • Groups of students Control factors • Tree’s prior height • Students’ prior test performance (usually most significant predictor) • Other factors beyond the gardener’s control: • Rainfall • Soil richness • Temperature • Other demographic characteristics such as: • Grade level • Gender • Race / Ethnicity • Low-Income Status • ELL Status • Disability Status • Section 504 Status

Another Visual Representation The Education Context Actual student achievement scale score Value. Added Starting student achievement scale score Predicted student achievement (Based on observationally similar students) Year 1 (Prior-test) Year 2 (Post-test)

15 Minute Break Any Questions or Comments?

Oak Tree Additions Expanding the Oak Tree Analogy Sean Mc. Laughlin

Oak Tree Additions 1. What about tall or short trees? � (high or low achieving students) 2. How does VARC choose what to control for? � (proxy measurements for causal factors) 3. What if a gardener just gets lucky or unlucky? � (groups of students and confidence intervals)

1. What about tall or short trees? (High or low achieving students)

1. What about tall or short trees? • If we were using an Achievement Model, which gardener would you rather be? • How can we be fair to these gardeners in our Value-Added Model? 93 in. Gardener D Gardener C 28 in. Oak C Age 4 Oak D Age 4

Why might short trees grow faster? • More “room to grow” • Easier to have a “big impact” Why might tall trees grow faster? • Past pattern of growth will continue • Unmeasured environmental factors How can we determine what is really happening? Gardener D Gardener C Oak C Age 4 Oak D Age 4

In the same way we measured the effect of rainfall, soil richness, and temperature, we can determine the effect of prior tree height on growth. The Effect of Prior Tree Height on Growth from Year 4 to 5 (inches) 40 30 in 9 in 35 30 25 20 Prior Tree. . . 15 10 5 0 0 20 40 60 80 100 120 Oak C Oak D Prior Tree Height (Year 4 Height in Inches) (28 in) (93 in)

Our initial predictions now account for this trend in growth based on prior height. • The final predictions would also account for rainfall, soil richness, and temperature. . in 0 3 + How can we accomplish this fairness factor in the education context? n. i 9 + Oak C Age 4 Oak C Age 5 (Prediction) Oak D Age 4 Oak D Age 5 (Prediction)

Analyzing test score gain to be fair to teachers Student 3 rd Grade Score 4 th Grade Score Abbot, Tina 244 279 Acosta, Lilly 278 297 Adams, Daniel 294 301 Adams, James 275 290 df Allen, Susan 312 323 Alvarez, Jose 301 313 Alvarez, Michelle 256 285 Anderson, Chris 259 277 Anderson, Laura 304 317 Anderson, Steven 288 308 Andrews, William 238 271 Atkinson, Carol 264 286 Test Score Range High Achiever Low

If we sort 3 rd grade scores high to low, what do we notice about the students’ gain from test to test? 3 rd Grade Score 4 th Grade Score Gain in Score from 3 rd to 4 th Allen, Susan 312 323 11 Anderson, Laura 304 317 13 Alvarez, Jose 301 313 12 Adams, Daniel 294 301 7 Anderson, Steven 288 308 20 Acosta, Lilly 278 297 19 Adams, James 275 290 15 Atkinson, Carol 264 286 22 Anderson, Chris 259 277 18 Alvarez, Michelle 256 285 29 Abbot, Tina 244 279 35 Andrews, William 238 271 33 Student Test Score Range High Low

If we find a trend in score gain based on starting point, we control for it in the Value. Added model. 3 rd Grade Score 4 th Grade Score Gain in Score from 3 rd to 4 th Allen, Susan 312 323 11 Anderson, Laura 304 317 13 Alvarez, Jose 301 313 12 Adams, Daniel 294 301 7 Anderson, Steven 288 308 20 Acosta, Lilly 278 297 19 Adams, James 275 290 15 Atkinson, Carol 264 286 22 Anderson, Chris 259 277 18 Alvarez, Michelle 256 285 29 Abbot, Tina 244 279 35 Andrews, William 238 271 33 Student Test Score Range High Low Gain High Low

What do we usually find in reality? Looking purely at a simple growth model, high achieving students tend to gain about 10% fewer points on the test than low achieving students. In a Value-Added model we can take this into account in our predictions for your students, so their growth will be compared to similarly achieving students.

Comparisons of gain at different schools before controlling for prior performance School A School B School C Student Population Advanced Proficient Basic Minimal High Achievement Artificially lower gain Medium Achievement Low Achievement Artificially inflated Why isn’t this fair?

Comparisons of Value-Added at different schools after controlling for prior performance School A School B School C Student Population Advanced Proficient Basic Minimal Fair

Checking for Understanding What would you tell a teacher or principal who said Value-Added was not fair to schools with: � High achieving students? � Low achieving students? Is Value-Added incompatible with the notion of high expectations for all students?

2. How does VARC choose what to control for? (Proxy measures for causal factors)

2. How does VARC choose what to control for? • Imagine we want to evaluate another pair of gardeners and we notice that there is something else different about their trees that we have not controlled for in the model. • In this example, Oak F has many more leaves than Oak E. • Is this something we could account for in our predictions? 73 in. Gardener F Gardener E Oak E Age 5 Oak F Age 5

In order to be considered for inclusion in the Value. Added model, a characteristic must meet several requirements: Check 1: Is this factor outside the gardener’s influence? Check 2: Do we have reliable data? Check 3: If not, can we pick up the effect by proxy? Check 4: Does it increase the predictive power of the model?

Check 1: Is this factor outside the gardener’s influence? Outside the gardener’s influence Gardener can influence Starting tree height Pruning Rainfall Insecticide Soil Richness Watering Temperature Mulching Starting leaf number Nitrogen fertilizer

Check 2: Do we have reliable data? Category Measurement Coverage Yearly record of tree height Height (Inches) 100% Rainfall (Inches) 98% Soil Richness Plant Nutrients (PPM) 96% Temperature Average Temperature (Degrees Celsius) 100% Starting leaf number Individual Leaf Count 7% Canopy diameter Diameter (Inches) 97%

Check 3: Can we approximate it with other data? ? Category Measurement Coverage Yearly record of tree height Height (Inches) 100% Rainfall (Inches) 98% Soil Richness Plant Nutrients (PPM) 96% Temperature Average Temperature (Degrees Celsius) 100% Starting leaf number Individual Leaf Count 7% Canopy diameter Diameter (Inches) 97%

Canopy diameter as a proxy for leaf count • The data we do have available about canopy diameter might help us measure the effect of leaf number. • The canopy diameter might also be picking up other factors that may influence tree growth. • We will check its relationship to growth to determine if it is a candidate for inclusion in the model. Gardener F Gardener E 33 in. Oak E Age 5 55 in. Oak F Age 5

If we find a relationship between starting tree diameter and growth, we would want to control for starting diameter in the Value-Added model. The Effect of Tree Diameter on Growth from Year 5 to 6 (inches) 40 35 30 ? 25 20 15 10 5 0 0 20 40 60 80 Tree Diameter (Year 5 Diameter in Inches) Tree Diameter

If we find a relationship between starting tree diameter and growth, we would want to control for starting diameter in the Value-Added model. The Effect of Tree Diameter on Growth from Year 5 to 6 (inches) 40 35 30 25 20 Tree Diameter 15 10 5 0 0 20 40 60 80 Tree Diameter (Year 5 Diameter in Inches)

What happens in the education context? Check 1: Is this factor outside the school or teacher’s influence? Check 2: Do we have reliable data? Check 3: If not, can we pick up the effect by proxy? Check 4: Does it increase the predictive power of the model?

Check 1: Is this factor outside the school or teacher’s influence? Outside the school’s influence School can influence At home support Classroom teacher English language learner status School culture Gender Math pull-out program at school Household financial resources Structure of lessons in school Learning disability Safety at the school Curriculum Prior knowledge Let’s use “Household financial resources” as an example

Check 2: Do we have reliable data? What we want • Household financial resources

Check 3: Can we approximate it with other data? What we want • Household financial resources What we have • Free / reduced lunch status Related data Using your knowledge of student learning, why might “household financial resources” have an effect on student growth? Check 4: “Does it increase the predictive power of the model? ” will be determined by a multivariate linear regression model based on real data from your district or state (not pictured) to determine whether FRL status had an effect on student growth.

What about race/ethnicity? Race/ethnicity causes higher or lower performance What we want What we have • General socio-economic • Race/ethnicity status • Family structure • Family education • Social capital • Environmental stress Related complementary data may correlate with one another (not a causal relationship) Check 4 will use real data from your district or state to determine if race/ethnicity has an effect on student growth. If there is no effect, it will not be included in the model.

What about race/ethnicity? If there is a detectable difference in growth rates We attribute this to a district or state challenge to be addressed Not as something an individual teacher or school should be expected to overcome on their own

Checking for Understanding What would you tell a 5 th grade teacher who said they wanted to include the following in the Value-Added model for their results? : A. B. C. D. 5 th grade reading curriculum Their students’ attendance during 5 th grade Their students’ prior attendance during 4 th grade Student motivation Check 1: Is this factor outside the school or teacher’s influence? Check 2: Do we have reliable data? Check 3: If not, can we pick up the effect by proxy? Check 4: Does it increase the predictive power of the model?

3. What if a gardener just gets lucky or unlucky? (Groups of students and confidence intervals)

3. What if a gardener just gets lucky or unlucky? Gardener G Oak A predicted growth: 10 inches Oak A Age 3 (1 year ago) Predicted Oak A

Gardener G Oak A actual growth: 2 inches For an individual tree, our predictions do not account for random events. Oak A Age 3 (1 year ago) Actual Oak A

Gardeners are assigned to many trees Gardener G Each tree has an independent prediction based on its circumstances (starting height, rainfall, soil richness, temperature)

How confident would you be about the effect of these gardeners? Gardener G Total trees assigned Trees that missed predicted growth Trees that beat predicted growth 5 3 2 Due to unlucky year Due to gardener Due to lucky year 1 2 0 2 Gardener H Total trees assigned Trees that missed predicted growth Trees that beat predicted growth 50 30 20 Due to unlucky year 7 Due to gardener Due to lucky year 23 15 5

Reporting Value-Added In the latest generation of Value-Added reports, estimates are color coded based on statistical significance. This represents how confident we are about the effect of schools and teachers on student academic growth. Green and Blue results areas of relative strength. Student growth is above average. Gray results are on track. In these areas, there was not enough data available to differentiate this result from average. Yellow and Red results areas of relative weakness. Student growth is below average.

3 Value-Added is displayed on a 1 -5 scale for reporting purposes. Grade 4 30 About 95% of estimates will fall between 1 and 5 on the scale. 3. 0 represents Numbers lower than meeting predicted Numbers higher than 3. 0 represent growth for your 3. 0 represent growth that did not meet students. that beat prediction. Since predictions are Students are learning Students are still based on the actual at a rate faster than learning, but at a rate performance of predicted. slower than students in your predicted. district or state, 3. 0 also represents the district or state average growth for students similar to yours.

3 READING Grade 4 30 3. 8 95% Confidence Interval Value-Added estimates are provided with a confidence interval. Based on the data available for these thirty 4 th Grade Reading students, we are 95% confident that the true Value-Added lies between the endpoints of this confidence interval (between 3. 2 and 4. 4 in this example), with the most likely estimate being 3. 8.

Confidence Intervals Color coding is based on the location of the confidence interval. The more student data available for analysis, the more confident we can be that growth trends were caused by the teacher or school (rather than random events). 3 READING Grade 3 13 4. 5 Grade 4 36 4. 5 Grade 5 84 4. 5 3 MATH Grade 3 13 1. 5 Grade 4 36 1. 5 Grade 5 84 1. 5

Checking for Understanding A teacher comes to you with their Value-Added report wondering why it’s not a green or blue result. She wants to know why there’s a confidence interval at all when VARC had data from each and every one of her students. (Don’t we know exactly how much they grew? ) 3 READING Grade 7 11 4. 2

Education Based Examples Attainment to Gain to Growth to Value. Added Ernest Morgan

Attainment and Gain Attainment – a “point in time” measure of student proficiency � compares the measured proficiency rate with a predefined proficiency goal. Gain – measures average gain in student scores from one year to the next

School performance from two lenses Gain Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Kidney transplant success rate (January 2005 to June 2007) UW-Health Expected Rate National Average 89. 78 92. 30 92. 56 Response to this loss of accreditation UW Hospital transplant doctors have said they are more aggressive than other centers in transplanting patients quickly, which can save more lives but lead to lower success rates. They have also said the hospital's organ donors are older and have more complications than elsewhere, but the formula to determine expected rates doesn't fully take that into account.

Growth – measures average gain in student scores from one year to the next � accounts for the prior knowledge of students. Next slide: Exercise #1 � Visual first � Questions on slide directly after

Growth: Starting Point Matters Reading results of a cohort of students at two schools 2006 Grade 4 Scale Score Average 2007 Grade 5 Scale Score Average A 455 465 10 B 425* 455* 30 School Grade 4 Proficient Cutoff 438 Grade 5 Proficient Cutoff 463 *Scale Score Average is below Proficient Example assumes beginning of year testing Average Scale Score Gain

Exercise #1: Discussion Questions Using only the 2006 Scale Score data, can we draw any conclusions about the performance of the two school cohorts in Grade 4? Using only the 2007 Scale Score data, can we draw any conclusions about the performance of the two school cohorts in Grade 5? What does an examination of gain – Average Scale Score - Grade 4 (2006) to Average Scale Score - Grade 5 (2007) – tell us about the performance of the two schools?

WKCE Pre-Test and Post-Test Grade 3 Summe r Nov Grade 4 Summe r Nov 3 rd Grade Value-Added Grade 5 Summe r Nov 4 th Grade Value-Added Grade 6 Nov 5 th Grade Value-Added Why don’t we have 8 th Grade Value-Added in Wisconsin?

Value-Added – measures average gain in student scores from one year to the next � accounts for the prior knowledge of students. � accounts for student demographic characteristics. � accounts for test measurement error.

What is Value-Added? It is a kind of growth model that measures the contribution of schooling to student performance on the WKCE in reading and in mathematics Uses statistical techniques to separate the impact of schooling from other factors that may influence growth Focuses on how much students improve on the WKCE from one year to the next as measured in scale score points

Demographic Controls Value-added controls for the demographic composition of schools These controls allow for fairer growth comparisons to be made Controlling for demographic factors make possible the measurement of differences in growth across demographic groups statewide (for example, ELL vs. non-ELL)

The Importance of Standard Measures Gas Station 1 87 Octane $2. 00/Gallon Gas Station 2 87 Octane $3. 00/Gallon With the information provided, from which station will you purchase gas? In making your decision, what are you assuming about the definition of a gallon?

Lunch (30 Minutes) Any Questions or Comments?

Value-Added Report Interpretation and Decision Making Anatomy of a Value-Added Report Value-Added Color Coding (Future Feature) Example Reports from Minnesota Project Modeling Interpretation of Estimates Sean Mc. Laughlin

What are you going to see today? Report based on growth during the 2009 -2010 school year Old grayscale report format

What are you going to see today? 2009 -2010 School-Level Value-Added in Reading and Math 2009 School. Level Percent Prof/Adv in Reading and Math Grade-Level Team Estimates (3 rd – 7 th)

What are you going to see today? Your school’s Value-Added and Percent Prof/Adv in Scatter Plot Form Other Schools in your District Value-Added in “Scale Score” or “Beat the Average”

What are you going to see today? Scatter Plots for Grade-Level Teams (3 rd – 7 th)

What are you going to see soon? Report based on growth during the 2010 -2011 school year Value-Added on 1 -5 “tier” scale New color report format

Confidence Intervals and Decision Making 3 MATH Grade 4 30 Grade 5 30 Overall 60 1. 3 2. 5 1. 9

Confidence Intervals and Decision Making 3 READING Grade 4 30 3. 0 95% Confidence Interval Value-Added estimates are provided with a confidence interval. Based on the data available for these thirty 4 th Grade Reading students, we are 95% confident that the true Value-Added lies between the endpoints of this confidence interval (between 2. 4 and 3. 6 in this example), with the most likely

Value-Added Color Coding 3 READING Grade 4 30 Grade 5 30 Grade 6 15 3. 0 2. 5 If the confidence interval crosses 3, the color is gray. 4. 1

Value-Added Color Coding 3 READING Grade 4 30 Grade 5 30 Grade 6 15 If the entire confidence interval is above 3, the color is green. 3. 7 4. 1 4. 4

Value-Added Color Coding 3 READING Grade 4 30 Grade 5 30 5. 1 Grade 6 15 5. 3 If the entire confidence interval is above 4, the color is blue. 4. 6

Value-Added Color Coding 3 READING Grade 4 30 Grade 5 30 Grade 6 15 2. 3 1. 8 1. 3 If the entire confidence interval is below 3, the color is yellow.

Value-Added Color Coding 3 READING Grade 4 30 Grade 5 30 0. 8 Grade 6 15 0. 3 1. 3 If the entire confidence interval is below 3, the color is red.

Value-Added Color Coding These colors are meant to categorize results at a glance, but making responsible decisions based on Value-Added estimates may require more careful use of the data. General guidelines: Green and Blue results areas of relative strength. Student growth is above average. Gray results are on track. In these areas, there was not enough data available to differentiate this result from average. Yellow and Red results areas of relative weakness. Student growth is below average.

Decision Making Examples To put this into context, let’s go through a few examples of decisions a school leader might make based on Value-Added results. 1. 2. 3. 4. 5. 6. Which grade-level teams should get additional help from a literacy coach? How might we pair up teaching teams in mentor relationships? When should I look outside my school for help at improving student learning? How can I prioritize resources if my results unclear? How do I interpret gray results, and what can I learn from them? Should I recommend professional development or a change in curriculum to particular teams? 7. Is Value-Added telling me a particular team is ineffective at teaching?

1. Which grade-level teams should get additional help from a literacy coach? 3 READING Grade 3 25 Grade 4 26 Grade 5 28 3. 0 3. 9 1. 8

1. Which grade-level teams should get additional help from a literacy coach? 3 READING 2 nd Priority Grade 3 25 3 rd Priority Grade 4 26 1 st Priority Grade 5 28 3. 0 3. 9 1. 8 This is a relatively low-stakes decision. A literacy coach may be beneficial to any of these teams. There is little risk in providing this resource to all the teachers. The limiting factor is likely to be availability of this resource. If possible, provide it to all teachers, but limited allocation may be based on area of most need.

2. How might we pair up teaching teams in mentor relationships? 3 MATH Grade 3 27 Grade 4 29 Grade 5 30 3. 4 1. 8 4. 1

2. How might we pair up teaching teams in mentor relationships? 3 MATH Grade 3 27 Pair with Grade 5 Grade 4 29 Pair with Grade 4 Grade 5 30 3. 4 1. 8 4. 1 Pairing up teaching teams with relative strengths with those of relative weaknesses may help your school’s ability to meet the needs of all students. In this case, grade 5 seems to be the strongest performer in math. If faced with a similar student population, the grade 4 team may be able to strategize with the grade 5 team to make more academic growth with the 4 th grade students. If 3 rd grade’s student population was more similar, this would be another pairing

3. When should I look outside my school for help at improving student learning? 3 READING Grade 3 39 Grade 4 42 0. 8 Grade 5 38 1. 9 1. 3

3. When should I look outside my school for help at improving student learning? 3 READING Grade 3 39 Grade 4 42 0. 8 Grade 5 38 1. 9 1. 3 In this situation, it appears that each of the three reading grade-level teams at this school is producing below-average growth with their students. This would be a good opportunity to seek out high performing teachers and grade-level teams in other schools in the same grade level throughout the district or state. Try to find schools serving students with similar achievement levels and other characteristics to maximize the chance that their high Value. Added strategies will be applicable to your students.

4. How can I prioritize resources if my Value-Added results are unclear? Past Academic Year 3 READING Grade 5 - Grade 6 35 Grade 7 13 Insufficient Data 2. 3 1. 7 If you could only provide a teaching coach to one grade at your school, which one would you choose?

4. How can I prioritize resources if my Value-Added results are unclear? Past Academic Year 3 READING ? Grade 5 - Grade 6 35 Grade 7 13 Insufficient Data 2. 3 1. 7 If you could only provide a teaching coach to one grade at your school, which one would you choose? Grade 5 has no information at all here. Grade 6 is yellow (below predicted), but just barely. Grade 7’s best estimate is 1. 7, lower than Grade 6. However, the color is gray due to a large confidence interval based on very few students (only 13). In cases like this, it is very important to look at other data. Let’s add in the Value. Added 3 year average to try to better understand the situation.

4. How can I prioritize resources if my Value-Added results are unclear? Past Academic Year 3 -Year Average 3 3 READING Grade 5 - Grade 6 35 Grade 7 13 Insufficient Data 2. 3 1. 7 3. 2 23 105 43 1. 9 3. 1 If you could only provide a teaching coach to one grade at your school, which one would you choose?

4. How can I prioritize resources if my Value-Added results are unclear? Past Academic Year 3 -Year Average 3 3 READING Grade 5 - Grade 6 35 Grade 7 13 Insufficient Data 2. 3 1. 7 3. 2 23 105 43 1. 9 3. 1 By considering the 3 year average, we can conclude: The 5 th grade team has a history of above average growth. The 6 th grade team is consistently producing below average student growth. The 7 th grade team seems to be on track now that we can see the historic data based on more students (43 instead of just 13). The 6 th grade team might benefit the most from the teaching coach. As always, consider context and other data. For example, do these teams have different class sizes, or is this just a difference in number of teachers / classrooms per

5. How do I interpret gray results, and what can I learn from them? 3 These three teams each have gray estimates. Would you interpret them the same way? READING Grade 3 52 Grade 4 12 Grade 5 19 2. 9 4. 7 3. 1

5. How do I interpret gray results, and what can I learn from them? 3 These three teams each have gray estimates. Would you interpret them the same way? READING Grade 3 52 Grade 4 12 Grade 5 19 2. 9 4. 7 3. 1 Grade 3 – The tight confidence interval around the gray estimate indicates we can be confident that this team’s Value-Added was close to average. Grade 4 – The best estimate of Value-Added is above average (4. 7). However, since it was based on a small amount of data (12 students), we cannot say with confidence that it was above average. This team may actually have below average Value-Added. Grade 5 – The best estimate is average Value-Added (3. 1). However, the wide confidence interval indicates that there was not enough data to rule out above or below average Value -Added.

5. How do I interpret gray results, and what can I learn from them? 3 These three teams each have gray estimates. Would you interpret them the same way? READING Grade 3 52 Grade 4 12 Grade 5 19 2. 9 4. 7 3. 1 As always, consider multiple data sources when making decisions. The 3 rd grade team has the most certain Value-Added estimate can be treated as one of the average teaching teams in the state. The 4 th and 5 th grade teams have less certain estimates and it is particularly important that additional sources of information are considered before making decisions about professional development, resource allocation, staffing

6. Should I recommend professional development or a change in curriculum to particular teams? 3 MATH Grade 3 41 Grade 4 42 Grade 5 44 2. 9 1. 1 3. 6

6. Should I recommend professional development or a change in curriculum to particular teams? 3 MATH ? Investigate Grade 3 41 Grade 4 42 Grade 5 44 2. 9 1. 1 3. 6 A below average Value-Added estimate for the 4 th grade teams indicates that during this past year, 4 th grade students at your school grew slower than predicted. Use other information and meet with this teaching team to determine root causes of this result. Is this a new content or grade area for one or more teachers on this team? Was there a particular challenge that this team faced last year? Work with the teachers to come up with a goal and plan for next year’s students. Consider instructional changes, classroom management, mentoring, standards

7. Is Value-Added telling me a particular team is ineffective at teaching? Past Academic Year 3 -Year Average 3 3 READING 3. 4 Grade 5 36 Grade 6 36 Grade 7 35 0. 7 1. 2 101 3. 2 100 0. 9 98 2. 8

7. Is Value-Added telling me a particular team is ineffective at teaching? Past Academic Year 3 -Year Average 3 3 READING ? Investigate 3. 4 Grade 5 36 Grade 6 36 Grade 7 35 0. 7 1. 2 101 3. 2 100 0. 9 98 2. 8 The 7 th grade team had the lowest Value-Added this year, but considering the 3 Year Average, this seems to be isolated to just the last year. Work with these teachers to determine why results may have been lower than usual last year. More concerning is 6 th grade Value-Added. Not only is last year’s performance low, but it has been consistently low over the 3 -Year Average. Consider what else you know about these teachers from observations and other data. Does Value. Added fit with a pattern of data showing low performance for this team? How might you and your school best support the 6 th grade team next year?

District Decision Making Examples What are potential district level decisions with current results? 1. Are there particular schools or groups of schools that require more support? • Are there trends based on particular challenges schools face? • Are there schools that “beat the odds”? How can we capitalize on their success? • How can we respond with resources, programs, and support structures? 2. Is there an overall weakness or strength in our teachers for a particular subjects? • How do we respond as a district?

1. Are there particular schools or groups of schools that require more support? Scenario 1 READING Percent Prof/Adv (2009) READING (relatively low-achieving district) 100 • 80 • 60 40 • 20 0 Are there trends based on particular challenges schools face? Are there schools that “beat the odds”? How can we capitalize on their success? How can we respond with resources, programs, and support structures? Schools in your district Schools in the state 1 2 3 4 5 READING Value-Added (2009 -2010)

1. Are there particular schools or groups of schools that require more support? Scenario 1 READING Percent Prof/Adv (2009) READING (relatively low-achieving district) 100 • What strategies are schools in group A using to meet the needs of low achieving students? • Can we replicate this success in group B? • Are there strategies our district can use to facilitate this improvement? 80 60 B 40 A 20 0 1 2 3 4 5 READING Value-Added (2009 -2010)

1. Are there particular schools or groups of schools that require more support? Scenario 2 MATH Grade 4 (relatively high-achieving district) MATH Percent Prof/Adv (2009) 100 80 60 40 20 0 Schools in your district Schools in the state 1 2 3 4 5 Grade 4 MATH Value-Added (2009 -2010)

1. Are there particular schools or groups of schools that require more support? Scenario 2 MATH Grade 4 MATH Percent Prof/Adv (2009) 100 (relatively high-achieving district) C D • What would you tell a principal in group C who said their Value-Added was low because their students had no room to grow on the test? • How can we learn from the success of group D and bring that knowledge to group C? • Are there programs or resources that group D is receiving that we could also provide to group C? 80 60 40 20 0 1 2 3 4 5 Grade 4 MATH Value-Added (2009 -2010)

2. Is there an overall weakness or strength in our teachers for a particular subjects? Scenario 3 READING Percent Prof/Adv (2009) READING Grade 5 (Consistently low Value-Added) 100 • How can we respond as a District? 80 60 40 20 0 Schools in your district Schools in the state 1 2 3 4 5 Grade 5 READING Value-Added (2009 -2010)

2. Is there an overall weakness or strength in our teachers for a particular subjects? Scenario 3 READING Percent Prof/Adv (2009) READING Grade 5 (Consistently low Value-Added) 100 • Can we connect to other districts across the state to collaborate? • Are some of our Network partners getting high Value. Added estimates? How can we learn from their success? 80 60 E 40 20 0 1 2 3 4 5 Grade 5 READING Value-Added (2009 -2010)

Translating what you see today into If the Confidence the new report format Interval crosses READING Percent Prof/Adv (2009) If the Confidence Interval is entirely Average “ 0” in the If the Confidence above “ 1 SD above current scale will 100 Interval is entirely average”, it will be blue become “ 3” on the new below “average”, it will scale One standard deviation be yellow 80 below average (the If the Confidence “average”, it will be Interval is entirely READING gray above average, it will be green One standard deviation above average (the edge of the gray bar) If the Confidence will become “ 4” on the Interval is entirely new scale Color is based entirely below “ 1 SD below on the Value-Added average”, it will be red Estimate and the Confidence Interval (Percent Proficient does not influence color) edge of the gray bar) will become “ 2” on the new scale 60 40 20 0 -9 -6 -3 0 3 6 9 READING Value-Added (2009 -2010)

Translating what you see today into the new report format READING Percent Prof/Adv (2009) READING 100 80 60 40 20 0 -9 -6 -3 0 3 6 9 READING Value-Added (2009 -2010)

Quadrant Exercise Quadrants 1 -4 Results of an Exploratory Study of Characteristics Ernest Morgan

128 Group Exercise: Quadrant Analysis Worksheets are in your binders

Purpose of Exercise 129 Examine Value-Added with the added dimension of Attainment Understand exactly what interpretations can and cannot be made regarding the data Discuss policy implications regarding the four quadrants of value-added and attainment data

Exercise Format 130 Divide into four sub-groups representing each of the four quadrants: � Quadrant 1 (High Value-Added, High Attainment) � Quadrant 2 (High Value-Added, Low Attainment) � Quadrant 3 (Low Value-Added, High Attainment) � Quadrant 4 (Low Value-Added, Low Attainment)

Exercise Format 131 For each quadrant discussion � 5 -10 minute sub-group discussion � Report from the sub-group representing the quadrant’s school team � Group discussion � Key Points

Quadrant Analysis 132 The mapping of value-added and attainment data in one graphic.

Quadrant Analysis 133 Mapping can be done using either the state or district as the reference point. � Data plots can be for districts (state value-added system), � or for schools, grades or classrooms (either a state or district value-added system)

Quadrant Analysis 134 Attainment data can be pulled from either the pre-test or the post-test. The pre-test is the preferred choice, since it is a pre-growth measure.

Quadrant Analysis 135 Perspectives of Policymakers � SEA analyzing districts and/or schools (statewide system) � Superintendent analyzing schools � Principal assessing school and analyzing grade- level performance

Quadrant Analysis 136 Plotting Value-Added data The following graphic addresses the nature of standard deviations and the application of quadrants to the data.

Value-Added Plotting Value-Added vs. Attainment – Quadrants Plus Standard Deviations

Quadrant Analysis 138 Various Representations � Quadrants Plus (Standard Deviation bars) � Nine Box

Value-Added vs. Attainment – Quadrants – Nine Box Plus 139

140 Quadrant 1 High Value-Added, High Attainment Be careful to accurately interpret the data. Characteristics � These are schools (could also be districts or grades) that are both high value-added and high attaining.

141 Quadrant 1 High Value-Added, High Attainment Questions 1. 2. 3. 4. Are all Quadrant 1 schools equal? What does it mean to be within one of the standard deviation bars? What does it mean to be within both of the standard deviation bars? While the standard deviation bars help to gauge the meaningfulness of the school position in the quadrant, what part of the story is still missing?

142 Quadrant 1 High Value-Added, High Attainment Quadrant 1: School Team Report

143 Quadrant 1 High Value-Added, High Attainment Key Points � It is critical that policymakers understand the dangers of over-interpreting the data. Expert Perspective � Michael Christian, Assistant Scientist, VARC

144 Quadrant 3 Low Value-Added, High Attainment The Ceiling Effect Myth Characteristics � Prior to value-added data, these high attaining schools had no reason to believe they were anything but highly successful. � In reality their high attainment has little to do with the impact of the school. � Some might refer to these schools as coasters, resting on the outside influences that lead to student attainment.

145 Quadrant 3 Low Value-Added, High Attainment Questions 1. 2. Why might the introduction of value-added to schools from this quadrant be a particularly painful process? Some would argue that because they are already high attaining schools, Quadrant 3 schools face a more difficult challenge in achieving high value-added status. Might there exist a “ceiling effect” that is curtailing their ability to be high value-added schools. ?

146 Quadrant 3 Low Value-Added, High Attainment Quadrant 3: School Team Report

147 Quadrant 3 Low Value-Added, High Attainment Key Points � There is no “ceiling effect. ” Expert Perspective � Brad Carl, Associate Researcher, VARC

148 Quadrant 3 Low Value-Added, High Attainment The Issue of Starting Point Scenario 1: if there is no difference in gaining points when starting from either 480 or 360, the slope of the regression line will be 1. Scenario 2: if it is easier to gain points when starting from 480 than 360, the slope of the regression line will be greater than 1. Scenario 3: if it is harder to gain points when starting from 480 than 360, the slope of the regression line will be less than 1.

149 Quadrant 2 High Value-Added, Low Attainment The Equitable Nature of Growth Characteristics � When only viewing attainment data these schools (could also be districts or grades) are deemed to be failing.

150 Quadrant 2 High Value-Added, Low Attainment Questions 1. 2. 3. Does attainment data alone paint an accurate picture of schools falling into Quadrant 2? If not, why not? Is there any benefit to studying the school culture and curriculum programs of schools from Quadrant 2? Is it enough to simply be a high value-added school? Can we expect Quadrant 2 schools to eventually become Quadrant 1 schools?

151 Quadrant 2 High Value-Added, Low Attainment Quadrant 2: School Team Report

152 Quadrant 2 High Value-Added, Low Attainment Key Points � It is critical that we recognize the success these schools are having regarding value-added. Expert Perspective � Lisa Geraghty, Outreach Specialist, VARC High Value-Added, Low Attaining Schools

153 Quadrant 4 Low Value-Added, Low Attainment The kids can’t learn. Characteristics � These are the most challenged of schools (could also be districts or grades), for they are neither high value-added nor high attaining.

154 Quadrant 4 Low Value-Added, Low Attainment Questions 1. 2. While value-added allows for an “apples to apples” comparison of schools (meaning that schools from any of the quadrants can be used as relevant examples), why might Quadrant 2 schools serve as a more palatable example for schools from Quadrant 4? From a Superintendent’s perspective, how might schools from Quadrant 4 be viewed regarding the use of district resources?

155 Quadrant 4 Low Value-Added, Low Attainment Questions (Continued) 3. Say a Principal has just reviewed the quadrant plot for her school and sees that the school has moved from a Quadrant 1 school last year to a Quadrant 3 or a Quadrant 4 school this year. How might looking at a school’s grade-level value-added plots be helpful in possibly diagnosing the change?

156 Quadrant 4 Low Value-Added, Low Attainment Quadrant 4: School Team Report

157 Quadrant 4 Low Value-Added, Low Attainment Key Points � Low attainment is no excuse for being low value- added. Regardless of starting point, all kids are capable of growth. Expert Perspective � Sara Kraemer, Associate Researcher, VARC Low Value-Added, Low Attaining Schools

Data - Your Reports Based on Growth During the Nov 2009 – Nov 2010 Period (Next Release Based on Nov 2010 – Nov 2011 Period) Ernest Morgan and Sean Mc. Laughlin

Your 2009 -2010 School Reports Sean and Ernie will circulate and answer questions about report interpretation. At 2: 45, we will regroup and wrap up for the day.

Feedback Discussion of Activities and Surveys Sean Mc. Laughlin

Feedback Form Thank you for helping us improve our sessions. What can we do to make this introduction easier to understand or more useful?

Question and Answer Questions or Comments? Exit Slips Leslie Steinhaus

Looking Ahead Conference Schedule Review March 6 th Agenda Leslie Steinhaus