Designing High Quality Affordable Assessment Systems Edward Roeber

Designing High Quality, Affordable Assessment Systems Edward Roeber Michigan State University National Research Council Board on Testing and Assessment April 6 -7, 2010 1

Overview n Balanced Assessment Systems n n n Summative Assessment Designs n n n n Horizontally Balanced Vertically Balanced Purpose of this Study Typical Current Assessment Program High Quality Assessment Program Development and Administration Costs Cost Reduction Strategies Results Recommendations 2

Vertically-Balanced System n Summative Component n n Interim Benchmark Component n n Instructionally-relevant, short-cycle summative assessments also using a broad array of assessment types Formative Assessment Component n n n Broad array of types of assessment Adequate professional development High-quality pre-service education All three components present, and equally important to the teacher and the student 3

Vertically-Balanced System n Portions of this system are out of balance: n n n n Interim benchmark assessments not thoughtfully created, acquired, or used Educators don’t understand how to use interim assessments Only conventional types of assessment used Lack of understanding about what formative assessment is and is not (e. g. , it is not an item bank) Lack of opportunities for educators to learn how to use formative assessment strategies as they teach Educators don’t learn about classroom and formative assessment in pre-service programs State summative assessments overpower the interim and formative assessments in the minds of educators (and students) 4

Horizontally-Balanced Systems n n n Multiple-choice items are the predominant form of assessment used by states Some states use constructed-response items Fewer still use performance assessments Emphasis is on speed of return of results, not quality or usefulness of the information once it gets there Schools feel compelled to “teach to the test” in some not-so-good ways - some subjects not taught at all and the rest are taught to the extent that they are on the test Only state-assessed skills are focused on in local assessments 5

Horizontally-Balanced Systems n This system is out of balance in some important ways n More types of items should be used in the system • Short- and extended-response items • Performance events • Performance tasks n n n This broader array of items could have a positive impact on teaching and learning - could be assessments worth teaching to These state assessments could also serve as a model for developing instructionally-sensitive interim assessments The prime question is can states can afford to use this broader array of assessment? 6

Purpose of This Cost Study n n Determine the cost of a typical state assessment program that primarily uses multiple-choice items Design a high quality summative assessment (HQA) as just described n n Note - HQA designed for Mathematics and Reading/Writing only. This excludes Science, alternate assessments for students with disabilities and ELPA assessments Determine the cost of such a HQA Determine whether several potential strategies could reduce costs of the HQA significantly Present various options and costs for an interim assessment system similar to the HQA 7

Current Typical Summative Assessment Design 8

Current Typical Interim Assessment Design 9

High Quality Summative Assessment Design 10

High Quality Interim Assessment Design 11

Development and Administration Costs n Typical Assessment Program - $20/Student n n HQA Summative Assessment Program - $55/Student n n Cost per student for the current typical assessment calculated using the ASG cost model (includes Year 0 development costs) Cost per student for the high quality assessment calculated assuming a single state implementation and no cost reduction strategies (includes Year 0 development costs) Most states cannot afford a nearly tripling of their state assessment costs, so the result will be a limit on innovative assessment types, unless something occurs 12

Cost Reduction Strategies n Participation in a consortium n n Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items n n n Model includes state consortium sizes of 10, 20, and 30 states Use of a state consortium reduces costs by an average of $15 per student Consortium approach represents a significant decrease in assessment cost Together, these innovations account for cost savings of about $3 to $4 per student Likely to account for more as efficiencies are developed in programming and using technology for these purposes Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring 13

Cost Reduction Strategies n Two different models for teacher-moderated scoring: n n Professional development model - no additional teacher compensation beyond that supported by the state or district for normal professional development days Stipend model - assume a $125/day stipend for teachers to score the performance items. • n n Note: teachers were assumed to score all performance items in a distributed scoring model These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid) Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21 14

Overall Cost Reduction Results 60 Diminishing Expenditures - per Student HQA Cost 50 40 Cost per Student 30 20 10 0 HQA 50/50 On-Line Testing Teacher Scoring HQA - All cost Distributed w/Stipend w/o Stipend variables Scoring on applied SCR/ECR Single State 10 State w/stipend 20 State 30 State HQA - All cost Current MC variables (Non-HQA Test) applied w/o stipend 15

Impact of Teacher Scoring Time on Costs 10 State Consortia - PE & PT Scoring Cost per Student - Impact of Increase in Scoring Time $ 45, 00 $ 41, 21 $ 40, 00 $ 36, 64 $ 32, 06 Cost per Student $ 35, 00 $ 27, 48 $ 30, 00 $ 22, 91 $ 25, 00 $ 18, 33 $ 20, 00 $ 15, 00 $ 10, 00 $ 13, 76 $ 13, 56 $ 15, 82 $ 18, 06 $ 20, 32 $ 22, 57 $ 24, 83 $ 27, 08 $ 5, 00 $Base 1 min 2 min 3 min 4 min 5 min 6 min Number of Minutes Added PE Cost per Student PT Cost per Student 16

Cost Study Conclusions n The development cost of a new HQA is relatively inexpensive relative to the total cost of the assessment n n n A key factor in determining whether states can adopt and sustain new improved assessments is ongoing administration costs In order to reduce costs, states should participate in an assessment consortium to share the overhead associated with development, administration, and management of assessments Larger consortia are somewhat more cost-effective n n The majority of cost savings relative to the single state case are seen even in at a 10 -state consortium size States should strongly consider being part of a large consortium where certain costs can be shared across many states, such as for item development and project management 17

Cost Study Conclusions n Implementing a HQA system with performance items is affordable, with teacher scoring of performance items at a price comparable to today’s assessments, when procured by a consortium of states n n n In order to implement and afford an HQA system that includes a variety of performance items, it is essential to have teachers involved in the scoring process The cost impact of increases in the time to score performance items is very significant The use of online technology (i. e. , online assessments) should be encouraged n n It has the potential to reduce assessment cost and improve quality The procurement of PCs to improve the student- to-PC ratio should be encouraged at all levels of the educational system 18

Recommendations n Developing and implementing an HQA will likely cost more than most current state assessments. It can be affordable for states if they look carefully at n n n the design of the summative assessment component finding a balance in the number of CR items, PEs, and PTs used consider various cost-reduction strategies It is recommended that state consortia go about the process of designing a new assessment in a thoughtful manner, then use a comprehensive costing model to analyze and determine the price in advance of any new assessment system they would like to implement State consortia interested in implementing a HQA should make sure they can afford the ongoing administration costs of the assessment before they embark on developing it 19

For More Information Edward Roeber Michigan State University 263 Erickson Hall East Lansing, MI 48824 (517) 432 -0427 roeber@msu. edu For a copy of the complete paper for this study, go to the Assessment Solutions Group website to download a copy: www. assessmentgroup. org 20