Understanding SBA Scores FROM THE ASSESSMENT DEVELOPMENT TEAM

Introduction ØPurpose: to answer some basic questions about Smarter Balanced Summative and Interim test scores ØAudience: DACs, principals, data/instructional coaches, teachers, parents ØShare: staff meetings, professional development, PLC meetings OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 2

Summative Test Scores For each content area (ELA/Literacy and Mathematics), Washington reports: § Overall scale score § Achievement level § Claim achievement category OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 3

Summative Test Scores Family Report/ Individual Student Report 6/13/2021 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 4

Summative Test Scores—scale scores OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 5

Summative Test Scores—scale scores Scale scores: • Are not determined only by a calculation of “raw-points-earned divided by total-points-possible. ” • Are related to the difficulty of the items a student was presented with. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 6

Summative Test Scores—scale scores Calli rice t a e B Anya Easy . . . tems i 2000 2500 ult Diffic items 3000

Summative Test Scores—achievement levels OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 8

Summative Test Scores—achievement levels OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 9

Summative Test Scores—achievement levels Example of ELA grade 7: OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 https: //www. smarterbalanced. org/assessments/scores/ 10

Summative Test Scores—achievement levels Educators and other stakeholders from Smarter Balanced member states participated in achievement level setting meetings in October 2014 to recommend cut scores. In December 2014, the state education chiefs voted to approve the cut scores for Smarter Balanced. Washington’s State Board of Education accepted these cut scores for Washington students in January 2015. More information about the process, and WA’s involvement can be found: ◦ https: //www. smarterbalanced. org/assessments/scores/ ◦ http: //www. k 12. wa. us/SMARTER/Achievement. aspx OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 11

Summative Test Scores—SEM Each student scale score is reported along with a Standard Error of Measurement value (SEM). This is the ± value displayed next to the 4 -digit scale score. This value can be different for each student. This value is dependent on the items a student saw, which ones were answered correctly, and those the student did not answer correctly. The SEM: • Indicates a range where the student’s score is likely to be if they took the test several times • Could also be called the “margin of error” or “standard error” OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 12

Summative Test Scores—SEM OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 13

Summative Test Scores—SEM An SEM is also included with state, district, school, and classroom/roster average scores in the Online Reporting System. The smaller the group of students, the larger the SEM. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 14

Summative Test Scores—SEM AVERAGE COMPARISONS STUDENTS State District School Classroom OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 15

Summative Test Scores--Claims OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 16

Summative Test Scores—Claims Claim reporting category results come from comparing the student’s claim scale score to the “standard” cut score, and incorporating the student’s SEM on the items in that claim. Step 1: Break the items up into their claim categories: Reading— 15 Writing— 13 Listening— 8 Research— 9 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 17

Summative Test Scores—Claims Step 2: Calculate the scale score for only the items in the claim category. This is called the claim scale score. Research Listening Writing Reading 2780 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 18

Summative Test Scores—Claims Step 3: Find the SEM for each of these claim scale scores. ± 50 These claim scale scores and claim SEM can be found in score files downloaded from WAMS and downloaded from ORS. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 19

Summative Test Scores—Claims David’s Reading claim scale score = 2780 Step 4: Get the “standard cut score” for the grade level and content area and do some math: standard cut score + (Claim SEM x 1. 5) = high standard cut score - (Claim SEM x 1. 5) = low 2552 + (50 x 1. 5) 2552 - (50 x 1. 5) 2552 + (75) 2552 - (75) 2627 2477 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 20

Summative Test Scores—Claims Step 5: Compare the answers from step 4 to the claim scale score. Below Standard low At/Near Standard high Above Standard high from #4 = low from #4 = 2627 2477 David’s Reading claim scale score = OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 2780 21

Summative Test Scores—Claims Step 5: Compare the answers from step 4 to the claim scale score. Below Standard low from #4 = 2477 At/Near Standard high from #4 = 2627 Above Standard David’s Reading claim scale score = 2780 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 22

Summative Test Scores—Claims Another way to illustrate: At/Near Standard Below Standard low Claim SEM x 1. 5 Above Standard high OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 23

Summative Test Scores—Claims The larger the SEM, the larger the “at/near” range: Below Standard At/Near Standard Claim SEM x 1. 5 Above Standard Claim SEM x 1. 5 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 24

Summative Test Scores—Claims Above Standard: student’s claim scale score > 1. 5 SEMs above the “standard” cut score Below Standard: student’s claim scale score > 1. 5 SEMs below the “standard” cut score At/Near Standard: student’s claim scale score within 1. 5 SEMs of the “standard” cut score OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 25

Summative Test Scores are not determined by a calculation of “raw-points-earned divided by totalpoints-possible” Scores are related to: üthe particular set of items the student saw, üthe difficulty of those items, üwhich of those items the student answered correctly, üthe standard error value (SEM) of the scale score, and üthe distance between the scale score and the “standard cut score. ” The achievement levels (1, 2, 3 and 4) are defined by Achievement Level Descriptors. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 26

Interim Comprehensive Assessment The ICAs are reported using the same three types of scores as the Summative: § Overall scale score § Achievement level § Claim achievement category The scores are determined the same way as the Summative scores described on the previous slides. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 27

Interim Assessment Block (IAB) Scores The IABs are reported using a “performance category”: ◦ Above ◦ At/Near ◦ Below This is very similar to the “claim achievement category” on the Summative. The IAB performance categories are determined the same way as described on the previous slides for the summative claim score categories: Raw score/item difficulty Scale score and SEM Compare to standard cut Performance category OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 28

Interim Assessment Block (IAB) Scores Raw score/item difficulty Scale score and SEM Compare to standard cut Performance category OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 29

Interim Assessment Block (IAB) Scores Percentage of points earned Scores are not determined by a calculation of “raw-points-earned divided by totalpoints-possible” OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 30

Interim Assessment Block (IAB) Scores Item difficulty across the content The IABs are groups of items about similar topics. IABs with harder content likely have more items with higher difficulty values. Raw points on one IAB topic cannot be compared to raw points on another IAB topic. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 31

Interim Assessment Block (IAB) Scores Standard Error of Measurement value (SEM) The smaller the number of items on a test, the greater the SEM. IABs with a small number of items (5 or 6) have the largest SEM. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 32

Now what? Look past the colors and labels Think about…. • Were there items that all your students struggled with? • Were there items that all of your students did well on? • Where are the outliers—items that all but a few students did well on? What instruction would benefit those few? • Were there trends in answers based on particular types of items? • What did you notice while hand scoring in THSS? OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 33

Now what? Remember: The IABs are one tool that can be used to gather evidence about a student’s understanding. The teacher should also gather additional evidence through formative practices. The IABs are not intended to be all the evidence that is collected, but they can be used to inform an educator’s thoughts about a student’s understanding. Several data points (including the IAB) should be used to formalize a conclusion about a student’s understanding. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 34

Interim Test Scores Interim Comprehensive Assessment scores are calculated the same way as the Summative Assessment scores. Interim Assessment Block performance categories are calculated the same way as the claim achievement categories. Focus on the items and patterns of student responses to those items. OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 35

Resources and reminders FAQ document posted along with this video on the WCAP Portal includes: § Links included in this presentation § Answers to frequently asked questions § Links to documents for further, more technical information Audience: DACs, principals, data/instructional coaches, teachers, parents Share: staff meetings, professional development, PLC meetings OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 36

Thank you! Please direct questions and comments about this presentation to: Kara Todd, Content Coordinator for Test Development kara. todd@k 12. wa. us Please direct content specific questions to: Anton Jackson, Mathematics Assessment Specialist anton. jackson@k 12. wa. us Shelley O’Dell, English Language Arts Assessment Specialist shelley. odell@k 12. wa. us OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6/13/2021 37