Growth Models Including a detailed description of the

Growth Models Including a detailed description of the Colorado Growth Model 1

Overview �Review Growth Model Options �Details on Colorado Growth Model �Major technical questions 2

Basic Definitions �Status describes the academic performance of a student or group (a collection of students) at a single point in time. �Growth describes the academic performance of a student or group (a collection of students) over two or more time points. �A growth model is a collection of definitions, calculations, and rules that quantifies student performance over two or more time points and supports interpretations about students, their classrooms, their educators, and their schools. 3

School-level “Growth” Measures �Year to Year changes in percent met �safe harbor �Grade to Grade changes in percent met or average scores �How are my fifth graders doing compared to my fourth graders �Cohort growth over time in percent met or average scores �How has my 2009 -10 third grade class performed overtime 4

Growth Models �A growth model is a collection of definitions, calculations, and rules that quantifies student performance over two or more time points and supports interpretations about students, their classrooms, their educators, and their schools. �Most growth models follow individual student growth, and these are the subject of discussion today. 5

Common Growth Models �Gain Scores �Trajectory �Categorical �Residual Gain �Projection �Student Growth Percentiles �Multivariate 6

Key Questions Growth Models �What interpretation does the growth model support � Formative? School accountability? Teacher evaluation? Program Evaluation? �What is the statistical foundation? � Gain-based, enhanced status, multivariate �What are required data features? � Vertical scale? Articulated cut scores? Large datasets? Factors beyond test scores? �What interpretations does the model support? � Does aggregation at the school level make sense? �Does the model include growth to standard? �What are common misinterpretations and threats to validity? 7

Gain Scores �Definition: Calculates the gain in a student’s test score from year to year. Example: 5 th grade math score – 4 th grade math score �Requires: vertical scale. �Strengths: very easy to calculate, very easy to aggregated �Weaknesses: may not be very reliable, easier to manipulate, poor vertical scale lessens value. �School measure: average gains �Oregon data: Gains vary by previous performance levels, see next slide. 8

Gain Scores – Oregon Data 9

Trajectory Model �Definition: If a student continues on this path, where will she be in the future (Oregon’s current model) �Requires: vertical scale useful, but not required �Strengths: Simple to compute �Weaknesses: assumes students make same growth year after year, targets for student already meeting may be difficult to determine �School measure: percent of students “on track” �Oregon data: see next slide. 10

Percent of students “On Track” 11

Categorical Model �Definition: evaluates students transitioning between performance levels �Requires: cut scores, benefits from articulated cut scores �Strengths: useful when vertical scale not available �Weaknesses: coarse measure of student progress �School measure: percent on track or weighted index of students improving their performance levels �Oregon data: none 12

Residual Gain Model �Definition: student gains are measured against “expected gains. ” This is an “enhanced status model. ” �Requires: state level data, model to predict expected gains, does NOT require vertical scaling �Strengths: value added measure, statistically robust many statistics can be derived from the measure (median, standard deviation, school variability) �Weaknesses: does not directly measure growth, growth to standard is an add-on �School measure: average residual gains. �Oregon data: gains are based on the mean gain, conditioned on the prior test score. See next slide. 13

Residual Growth versus Status Residual Gain and Status – 2009 -10 100, 0 90, 0 80, 0 Percent Met 70, 0 60, 0 50, 0 40, 0 30, 0 20, 0 10, 0 (8, 0) (6, 0) (4, 0) (2, 0) 0, 0 Average Residual Gain 2, 0 4, 0 6, 0 14

Projection Model �Definition: uses a regression model to predict future scores. (Oregon’s federal growth model pilot submission was a hybrid of this model and a trajectory model) �Requires: linear regression model, state level data. �Strengths: empirically grounded: uses historic data to help predict current student’s performance, can use data from multiple subjects, supports growth to standard �Weaknesses: assumptions, such a linear growth, are often untenable. �School measure: average predicted score, percent of students “on track” �Oregon data: I’ve looked at versions of this, but don’t have pretty data to present. 15

Student Growth Percentiles �Definition: student growth compared to other students with similar score histories, expressed as a percentile. It is an “enhanced status model”. �Requires: state data, large data sets, does NOT require vertical scale �Strengths: includes growth to standard algorithm, is value added �Weaknesses: complex model �School measure: median or mean growth percentile. �Oregon data: see next slide and later in this presentation. 16

Growth Percentiles in Oregon Growth Percentiles - Reading Grade 6 for 2009 -10 25 20 Gain in 6 th Grade 15 10 5 0 191 196 201 206 211 216 221 226 231 236 241 246 251 256 261 -5 -10 -15 5 th Grade Reading Scores pct 10 pct 25 pct 50 pct 75 pct 90 17

The Colorado Growth Model �Basic Assumptions �Sample Data �Adequate Growth �Major technical decisions 18

Quick Description �Students with two consecutive years of test scores are provided with a growth percentile. �Student growth is compared to “academic peers”, who are students with similar score histories. �A student growth percentile of 60 means the student “grew” as much or more than 60 percent of students with similar test score histories. �Various percentiles of growth can be projected into the future to determine “Adequate Growth Percentiles” 19

Glossary �Academic Peers – students with the same or similar test scores in the past. �Student Growth Percentile (SGP) – the growth of the student as compared to her academic peers �Adequate Growth Percentile (AGP) – the growth percentile that needs to be sustained in order for the student to meet or still meet standard in 3 years. 20

Growth Percentiles in Oregon Growth Percentiles - Reading Grade 6 for 2009 -10 25 20 Gain in 6 th Grade 15 10 5 0 191 196 201 206 211 216 221 226 231 236 241 246 251 256 261 -5 -10 -15 5 th Grade Reading Scores pct 10 pct 25 pct 50 pct 75 pct 90 21

Sample SGP Output Prior Grade Current Grade Prior RIT Current Growth RIT 5 th 6 th 195 200 5 th 6 th 209 5 th 6 th 5 th SGP AGP 5 16 80 208 -1 8 72 209 214 5 34 64 6 th 209 218 9 61 58 5 th 6 th 209 222 13 85 53 5 th 6 th 209 226 17 96 48 5 th 6 th 227 232 5 57 27 22

Student Growth Report Student reports can include the following: �Student test history �Current Growth percentile – is the growth low, average, or high �Scores that correspond to 1 st, 35 th, 65 th and 99 th percentile (for comparison) �Projections into the future for various growth percentiles 23

Sample Student Report 24

Aggregations at School Level �Median SGP by subgroup �This is the recommendation by the SGP author, though others recommend mean �Median AGP by subgroup �This is the recommendation of the SGP author, though others recommend mean �Subgroups Making Adequate Growth �Comparison of the median SGP and the median AGP. �Percent meeting AGP �This is not in the proposed school ratings model. 25

Draft School Level Growth Report Median SGP Median AGP Subject Subgroup 2011 -12 2012 -13 Combined Math All Students 45 51 47 34 37 36 Math LEP 51 54 53 56 61 58 Reading All Students 55 52 53 26 28 27 Reading LEP 45 48 47 65 63 64 26

Webbased reporting 27

Now the Complications �SGPs -- The model can take into account more than two years of data for each student �By using more than two years of data we may get a better estimate of the student’s growth. �See the next slide for examples 28

SGPs – Higher Order Growth This data show 3 years of test scores can affect growth percentiles. Current Back 2 Grade years RIT Prior RIT Current Growth RIT SGP 7 th 225 231 6 47 Adjust ment 7 th 206 225 231 6 77 +30 7 th 211 225 231 6 69 +22 7 th 216 225 231 6 57 +10 7 th 221 225 231 6 45 -2 7 th 226 225 231 6 33 -14 7 th 231 225 231 6 17 -30 29

Focus and Priority Schools �New “rating” system that will identify focus and priority schools. �Growth is a very strong factor in the rating. Draft Focus/Priority Identification Weighting Elem/Middle High Achievement 25% 20% Growth 50% 20% Subgroup Growth 25% 10% Graduation -- 35% Subgroup Graduation -- 15% 30

Contact Information and Links �Details on the waiver: http: //www. ode. state. or. us/go/nextgen �Details on priority, focus, and model schools: http: //www. ode. state. or. us/search/page/? id=3742 �Priority, Focus and Model School Detail Sheets: http: //www. ode. state. or. us/search/page/? id=3749 �Policy and Technical Manual: http: //www. ode. state. or. us/wma/policy/accountabilit y/nextgen 2012/nextgenaccountabilitymanual 2012. pdf �Jon Wiens � Email: jon. wiens@state. or. us 31