Vertical articulation of content and performance standards Results

Vertical articulation of content and performance standards Results from a survey of the State Departments of Education Priya Kannan Educational Testing Service Presented at the 2014 annual National Conference on Student Assessment (NCSA) June 24 -27, 2014, New Orleans, LA Copyright © 2014 by Educational Testing Service. 1

Context § § Post-NCLB requirements to report AYP § Administer tests and report AYP in Grades 3 – 8 + HS § What have states been doing? - (apriori and post-hoc adjustments) Common core state standards and the nextgeneration assessments § What are states’ future plans? 2 Copyright © 2014 by Educational Testing Service.

Online Survey § To understand current state practices in recommending and articulating content and performance standards across the grade levels § Focus on Grades 3 – 8 ELA and Math § Assessment Directors or another recommended delegate (one respondent from each state) § 35 states responded (26 states completed all sections) § Responses between October 2013 and January 2014 Copyright © 2014 by Educational Testing Service. 3

Respondent characteristics CURRENT POSITION: § ~ 63% had spent between 1 and 6 years § ~ 18% had spent over 10 years § Almost all of the respondents (~94%) reported being involved in the planning and implementation of the standard-setting meetings for their state assessments Director / Chief of Student Assessments & Accountability 3% 12% Assistant Director / Deputy Superintendent 16% 69% Copyright © 2014 by Educational Testing Service. Psychometrics Director / Psychometric Manager Other 4

Content standards § Common Core State Standards (CCSS) initiative (2009) § § Push toward providing a consistent and clear understanding of what students are expected to learn at the forefront Research-based holistic approaches to curriculum development and assessment design § Frameworks (e. g. , Bejar, Braun, & Tannenbaum, 2007) that integrate educational policy, learning theory, and curricular considerations Copyright © 2014 by Educational Testing Service. 5

Content standards (Grade 3 – 8) Use of learning progressions in the development of content standards 9% Developed based on an underlying learning progression for both ELA and Math 6% Developed based on an underlying learning progression for ELA, but not for Math 76% Not developed based on an underlying learning progression 6 Copyright © 2014 by Educational Testing Service. NOTE: Some states did not respond (will not add to 100%)

Content standards (all grades) 30 25 73 % 20 15 52 % 55 % 40 % 10 5 0 Content standards for EOC articulated to Grades 3 -8 Content standards for all grades developed to measure a progressively deeper understanding of concepts at subsequent grades EOC standards are designed to ensure that students graduating are "college and career ready" at the end of high school Content standards for grades 3 -8 designed to ensure that students are "on-track" 7 Copyright © 2014 by Educational Testing Service. NOTE: States were allowed to choose more than one response (will not add to 100%

Alignment to the common core States' plans for aligning content standards for ELA and Math to the Common Core State Standards 3% 9% Already adopted the CCSS Conducted an alignment study 18% 70% Plan on conducting an alignment study Do not plan on adopting the CCSS 8 Copyright © 2014 by Educational Testing Service.

Personalization § Modifications to the common core: § Added content to the CCSS that are predominantly “stepping stone” standards and enhance the specificity of grade-level standards § Added to or edited their existing content standards to align to the CCSS § Did not officially adopt the CCSS, but development of their state content standards were informed by the CCSS 9 Copyright © 2014 by Educational Testing Service.

Standard setting § Standard-setting meeting for their grades 3 – 8 ELA and Math: § Most states reported having conducted a meeting between 2009 and 2010 § Most states reported that meetings for all grades in each content area were conducted in the same year, within related meetings § In some cases, meetings were staggered so that performance standards were set as grade-level assessments became operational Copyright © 2014 by Educational Testing Service. 10

Popular Standard-setting methods The Bookmark method was used by 74% of the states for Grades 3 – 8 ELA and Math Item Descriptor matching method Modified Angoff method § In addition, one or more of these other methods were used either in combination with the Bookmark particularly for specific grades and/or subjects 8% 19% 15% Body of work / Performance profile method(s) Contrasting / Borderline group method(s) 12% 15% 12% Statistical interpolations / extrapolations Using empirical evidence and external benchmarks Copyright © 2014 by Educational Testing Service. 11 NOTE: States were allowed to choose more than one method (will not add to 100%)

Cross grade discussions 20 Cross grade panelist representation on grade-level panels 18 55 % Number of respondents 16 Grade-level presentation of cutscores recommended in adjacent (or all) grades 14 12 10 8 6 4 30 % 24 % 18 % There was no cross grade discussion during the standard setting meeting 2 0 Cross grade discussion (either with all panelists or tables leaders) after the rounds of judgment are completed Cross-grade discussion implementation 12 Copyright © 2014 by Educational Testing Service. NOTE: States were allowed to choose more than one method (will not add to 100%)

Vertical articulation of performance standards § § Achieve cross grade consistency in the rigor of cut-scores § Assess student progress across the grade-level PL categories § Predict which students will attain a required criterion PL (proficient / on-track to CCR) in subsequent grades Some of the popular methods used among states are discussed here… § Predominantly states reported using a combination of methods Copyright © 2014 by Educational Testing Service. 13

Cut-score articulation methods § Impact percentage smoothing -- percentages of examinees that would be classified into each performance level across the grades § A reactor panel at a standards validation meeting composed of policy makers, TAC, or grade-level panel representatives § A joint (meta) panel composed of panel representatives from grade-level meetings Copyright © 2014 by Educational Testing Service. 14

Cut-score articulation methods § Statistical interpolation -- Cut-scores for the end grades (e. g. , Grades 3 and 8) are recommended based on a panel meeting. The cut-scores for intermediary grades (e. g. , Grades 5, 6, 7) are derived through statistical interpolation. § Vertical scale-based methods -- Concurrent or chained calibrations of linked test items across the grade levels may be used to estimate examinee ability on a single underlying continuum Copyright © 2014 by Educational Testing Service. 15

Cut-score articulation methods § Benchmarking -- Benchmarks are obtained for the high school assessments based on external criteria such as performance on: college courses, college entrance exams, national and international assessments § Vertical scales are used to articulate at the same scale point using concurrent calibrations for the lower grade levels § Logistic regressions are performed to predict the point on the adjacent lower grade where students have a said probability (e. g. , Copyright 66%)© 2014 ofbyachieving the cut at the higher Educational Testing Service. 16

Cut-score articulation methods (Grades 3 – 8) Impact percentage smoothing: panel representatives Impact percentage smoothing: standards validation meeting Statistical interpolations 14 Number of respondents 12 10 8 6 4 48 % 37 % 26 % Based on a vertical scale 22 % 2 19 % 22 % 15 % 0 Method Benchmarking for high school and articulated to lower grades Other Not articulated 17 Copyright © 2014 by Educational Testing Service. NOTE: States were allowed to choose more than one method (will not add to 100%)

Cut-score articulation (Grades 3 -8 to EOC) Cut-scores for EOC articulated to grade 3 -8 16 Number of respondents 14 56 % 12 10 Cut-scores for grades 3 -10 were established in the same meeting 37 % 8 Cross-grade secondary committee that reviewed all preliminary and smoothed cuts 6 4 2 11 % 15 % 11 % 0 Method Benchmarked cut-scores for EOC used to predict if students at the lower grade levels are "on-track" 18 Copyright © 2014 by Educational Testing Service.

Challenges reported § Panelists have trouble separating policy implications from the task at hand (what students know and are be able to do based on the PLDs) § When articulation does not result in a smooth trajectory § § Option of adopting a transitory cut-score until the entire process is revisited § Political and practical implications of artificially smoothing Up-and-down patterns (wherein the cut-scores for some grades are considerably more rigorous than others) cannot be completely alleviated Copyright © 2014 by Educational Testing Service. 19

Plans for next-generation assessments § Further explore the development of assessments based on vertical scales § Explore predictive-validity based approaches to articulating cut-scores § Majority of the states reported that they are depending on the consortia (e. g. , PARCC, SBAC) to come up with the answers 20 Copyright © 2014 by Educational Testing Service.

Key findings § 94% of the states report developing articulated content standards based on an underlying progression § 74% of the states report using the Bookmark method. § 82% of the states reported having some form of a cross-grade discussion. § 78% of the states reported using various methods to articulate the cutscores. § Impact percentage smoothing was most frequently used (66% of states). Questions / Comments? § Predictive and evidence-based methods seem to be gaining in Please e-mail me (pkannan@ets. org) for a copy popularity. the report. Copyright © 2014 by Educational Testing Service. of 21