Reciprocal accountability Using local and state assessment data

Reciprocal accountability: Using local and state assessment data to produce annual accountability determinations Scott Marion National Center for the Improvement of Educational Assessment CCSSO: National Conference on Student Assessment San Diego, CA June 22 -24, 2015

Beginning to Address Challenges of Assessment of CBE Marion_Reciprocal Accountability_CCSSO_June 24, 2015 2

Comprehensive Assessment Systems • When done well, competency-based assessment systems can be instantiations of comprehensive assessment systems – Multiple types of assessments, e. g. , summative, interim, unitbased, formative – Multiple loci of control—classroom, school, district, and state • What is the role of local curriculum (not standards) in CBE-based assessment design? • In many (most? ) cases, comprehensive assessment systems require state and local cooperation • PACE’s reciprocal system requires tight coherence among the state, collaborative, and local assessment systems Marion_Reciprocal Accountability_CCSSO_June 24, 2015 3

Key Technical Challenges/Requirements for PACE 1. Reporting achievement (status) – In the language of USED, “annual determinations” 2. 3. 4. Calculating and reporting “growth” Defining, evaluating, and monitoring comparability Ensuring and enhancing equity • Focus today on producing comparable, annual determinations… Marion_Reciprocal Accountability_CCSSO_June 24, 2015 4

Comparability • A lot of energy focused on comparability over the past five years (e. g. , PARCC-Smarter Balanced) • Psychometricians worry a lot about the differences among things like equating, concordance, projection… – Congressional staff now worried about differences among “equal, ” “equivalent, ” and “comparable” • The U. S. has a particular obsession with “interchangeability, ” but we always need to ask ourselves comparable for what • We also need to learn from other countries and some enlightened measurement specialists in the US Marion_Reciprocal Accountability_CCSSO_June 24, 2015 5

Comparability Challenges in a Reciprocal System Smarter Balanced in three grades PACE Common Performance Task PACE Competency 1 Local performance assessments Comparable Annual Determinations Competency 2 Local performance assessments Competency 3 Local performance assessments District-Level Competency Scores And so on Marion_Reciprocal Accountability_CCSSO_June 24, 2015 6

Focus on Student Work • Ultimately, comparability will be evaluated and achieved through intense interrogation of student work • Multiple layers of the process – – Within school calibration and comparable scoring Development of common PLDs closely linked to Smarter Balanced Across district comparability analyses “Standard setting” Marion_Reciprocal Accountability_CCSSO_June 24, 2015 7

Cross-District Comparability • Student work samples are used to evaluate the comparability of scoring (and, by inference, expectations) across districts – The goal is not to repeat the inter-rater reliability analyses conducted at the local levels • Rather, the plan is to evaluate prototypical and borderline papers from each district to judge the extent to which students are held to common expectations • For example…. Marion_Reciprocal Accountability_CCSSO_June 24, 2015 8

Matching Student Work to PLDs Marion_Reciprocal Accountability_CCSSO_June 24, 2015 9

Standard Setting • Initial achievement levels need to be established withindistrict – Unique assessments for determining competency • Comparability analyses will help calibrate differences in standards (e. g. , proficiency) across districts • Profile approach will be used with the student work generated from the common tasks as anchors – Smarter Balanced results will be used as key benchmark data – Next year, we will have work samples collected throughout the year Marion_Reciprocal Accountability_CCSSO_June 24, 2015 10

Some additional resources Domaleski, C. , Gong, B. , Hess, K. , Marion, S. , Curl, C. , Peltzman, A. (2015). Assessment to support competency-based pathways. Washington, DC: Achieve. www. Achieve. org and www. nciea. org Marion, S. F. & Buckley, K. (in press). Design and implementation considerations of performance-based and authentic assessments for use in accountability systems. In Braun, H. (ed). Meeting the Challenges to Measurement in an Era of Accountability. Washington, DC: NCME. Marion, S. (2015, Feb). Two sides of the same coin: Competency based education and Student Learning Objectives. Published by Competency Works. http: //www. competencyworks. org/resources/two-sides-of-thesame-coin-competency-based-education-and-student-learningobjectives/ Marion, S. , & Leather, P. (2015). Assessment and accountability to support meaningful learning. Education Policy Analysis Archives, 23(9). http: //dx. doi. org/10. 14507/epaa. v 23. 1984 Mc. CLarty, K. L. & Gaertner, M. (2015). Measuring mastery: Best practices for assessment in competency-based education. Washington, DC: American Enterprise Institute Marion_Reciprocal Accountability_CCSSO_June 24, 2015 11

For more information: Center for Assessment www. nciea. org Scott Marion smarion@nciea. org Marion_Reciprocal Accountability_CCSSO_June 24, 2015 12

Discussion Questions 1. What are the opportunities and challenges associated with implementing reciprocal accountability in your state? – Technical? – Policy? – Practical? 2. What should be the state’s role in a reciprocal system for ensuring that all students are being held to comparable expectations and being provided equitable opportunities to learn? Marion_Reciprocal Accountability_CCSSO_June 24, 2015 13