ISTAR Standard Setting Overview of ISTAR Standard Setting

ISTAR Standard Setting

Overview of ISTAR Standard Setting Methodology • Item Descriptor (ID) Matching was suggested by the Indiana TAC for finding cut scores on the ISTAR tests. • ID matching process: • Evaluative process using an Ordered Item Booklet (OIB) and Performance Level Descriptors (PLDs) over several rounds • Based on the Performance Level Descriptors, panelists made judgments about each of the items in the OIB, classifying it as Developing, Meeting, or Exceeding proficiency (Rounds 1 and 2) • Panelists then indicated a “threshold region” where the item descriptors change from mostly lower PLD to mostly higher PLD (Rounds 1 and 2) • Within the threshold region, panelists are asked to find a transition point in the OIB where knowledge, skills, and cognitive processes required by items change from the lower PLD and begin to match more closely the knowledge, skills, and cognitive processes described in the higher level PLD (Rounds 3 and 4)

ISTAR Standard Setting • Meeting and Logistics: • General session — overview of ISTAR, introduction to the process, meeting logistics, housekeeping, security protocols • Experience the assessment — panelists utilized the testing environment mirroring the student experience and interaction with items • PLD discussion — panelists reviewed and discussed the PLDs for the upper grade — what knowledge, skills, and cognitive processes expected of students classified in each performance level • OIB discussion — panelists reviewed and discussed the OIB for the upper grade — what knowledge, skills, and cognitive processes are required to answer it correctly, and why is each item more difficult than the previous ones in the OIB • ID Matching practice round — panelists were assigned sample (10) items from the OIB to practice the tasks of matching items to PLDs

ISTAR Standard Setting • Meeting and Logistics (cont. ) • Rounds of judgments and feedback • 4 rounds of judgments (assign every item in the OIB to a PLD, identify threshold regions, and locate a cut score within each threshold region) • After each round, facilitator provided panelists with data that they can use to evaluate and inform their judgments in the subsequent rounds. Feedback data varied by round but generally included summary statistics (i. e. , descriptive statistics, such as the mean, median, minimum, and maximum page numbers) for the threshold region boundaries and recommended cut scores, agreement data, and impact data. • Repeat process with the lower grade-level assessment • Table Leaders continued to the Vertical Articulation meeting • Panelists examined the reasonableness of the proposed cut score recommendations across the grade levels

ISTAR Standard Setting Logistics • Standard Setting Meetings: • June 19– 21: ELA and Science • June 26– 28: Mathematics and Social Studies • Panels consisted of multiple breakouts: • Four grade band panels (3– 4, 5– 6, 7– 8 and 10) ELA and Mathematics • Two grade band panels (4 and 6, and 10) Science • One grade band panel (5 and 7) Social Studies • Each grade band panel had a facilitator and a content specialist • Each panel consisted of 8– 9 Indiana Educators in two tables; most panels had one general education teacher • Two of the panelists were designated by IDOE as table leaders — they led table discussions, took notes, and spoke on behalf of the table during the large-group discussion

ISTAR ELA Reconvening • Questar identified an issue in early July with the impact data presented to the ISTAR ELA panelists. • TAC recommended a cut score validation to ensure the recommendations made by the panelists during the June meeting reflected a valid process. • A subset of panelists reconvened on August 7, 2017 to consider judgments for Round 4 and vertical articulation with accurate data confirmed by a third party vendor.

ISTAR Performance Levels • Two cuts were recommended for each grade and subject: • The Meeting Proficiency cut — The cut score that differentiates Developing Proficiency performance from Meeting Proficiency performance. • The Exceeding Proficiency cut — The cut score that differentiates Meeting Proficiency performance from Exceeding Proficiency performance.

ISTAR Standard Setting Results • Tables and graphs in the coming slides present the impacts of cut scores • Multiple impacts are given for all grades and content areas • Impacts of cuts recommended in Round 4 by grade band panels • Impacts of cuts adjusted in vertical articulation committees, if adjustments were made • Impact of IDOE recommendation for ELA and Mathematics • Impacts of cuts on the whole population which includes the No Mode of Communication students reported as “Undetermined” proficiency level

English Language Arts

ISTAR Standard Setting Results – ELA, Round 4 Impact Data for ISTAR ELA after Round 4 120% 100% 80% 12 25 28 29 20 21 26 18 60% 40% 26 48 36 36 31 37 70 47 20% 46 32 43 43 27 0% Grade 3 Grade 4 Grade 5 Developing Proficiency Grade 6 Meeting Proficiency Grade 7 Exceeding Proficiency Grade 8 Grade 10

ISTAR Standard Setting Results – ELA, Vertical Articulation Impact Data for ISTAR ELA after Vertical Articulation 120% 100% 80% 12 25 28 29 20 21 26 18 60% 40% 26 48 36 36 31 37 70 47 20% 46 32 43 43 27 0% Grade 3 Grade 4 Grade 5 Developing Proficiency Grade 6 Meeting Proficiency Grade 7 Exceeding Proficiency Grade 8 Grade 10

ISTAR Standard Setting Results – ELA, All Students Impact Data for ISTAR ELA after Vertical Articulation, NMC Students Included 100 90 80 11 24 24 26 26 19 20 25 16 35 70 60 35 45 29 50 40 35 64 30 43 43 41 31 20 40 26 10 0 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Possible adjustments to ELA cuts • Grade 3 • Currently the “meeting proficiency” cut is set on item 10 of the OIB, resulting in 70% of students classified as “developing proficiency. ” • Vertical articulation panel would support moving the cut to item 7, as they thought items 7, 8, and 9 were also better aligned to “meeting proficiency” than “developing proficiency. ” • Grade 6 • Currently the “meeting proficiency” cut is set on item 8 of the OIB, resulting in 32% of students classified as “developing proficiency. ” • Moving to item 10 is within the threshold range of panelist ratings and creates better alignment with the other grade levels.

Possible adjustments to ELA cuts (cont. ) • Grade 10 • Currently the “exceeding proficiency” cut is set on item 34 of the OIB, resulting in 36% of students classified as “exceeding proficiency. ” • Vertical articulation panel said they could support moving the cut to item 37, as it was the point where items clearly transitioned from “meeting proficiency” to “exceeding proficiency. ”

ISTAR Standard Setting Results – ELA, Grades 3, 6, 10 adjusted Impact Data for ISTAR ELA, Grades 3, 6, 10 Adjusted 100% 90% 12 26 25 28 29 20 21 26 80% 70% 32 27 60% 44 36 31 50% 42 40% 30% 61 47 20% 46 36 43 43 27 10% 0% Grade 3 Grade 4 Grade 5 Developing Proficiency Grade 6 Meeting Proficiency Grade 7 Exceeding Proficiency Grade 8 Grade 10

ISTAR Standard Setting Results – ELA, all students, Grade 3, 6, 10 adjusted Impact Data for ISTAR ELA, NMC Students Included, Grades 3, 6, 10 adjusted 100 90 80 11 24 24 26 26 19 20 25 16 35 70 60 35 45 29 50 40 35 64 30 43 43 41 31 20 40 26 10 0 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ELA: Context for Results • Specifically, the Vertical Articulation panel had the following thoughts about grade 3 impact data: • PLDs and the associated assessment were seen as being more difficult in this grade than in other grades • Grade 3 is the first year the students take the assessment, so they might be struggling with the concept of test-taking

ELA Scale Score Cuts Meeting Proficiency Exceeding Proficiency Grade 3 351 382 Grade 4 355 385 Grade 5 354 391 Grade 6 348 388 Grade 7 343 378 Grade 8 351 380 Grade 10 344 386

Mathematics

ISTAR Standard Setting Results – Math, Round 4 Impact Data for ISTAR Math after Round 4 100 90 12 80 70 32 13 20 9 25 6 3 4 11 31 5 25 38 60 50 85 40 30 56 67 66 63 Grade 4 Grade 5 Grade 6 72 57 20 10 0 Grade 3 Developing Proficiency Meeting Proficiency Grade 7 Grade 8 Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – Math, Vertical Articulation Impact Data for ISTAR Math after Vertical Articulation 100 90 12 80 70 32 13 20 9 25 6 3 4 11 31 5 25 38 60 50 85 40 30 56 67 66 63 Grade 4 Grade 5 Grade 6 72 57 20 10 0 Grade 3 Developing Proficiency Meeting Proficiency Grade 7 Grade 8 Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – Math, All Students Impact Data for ISTAR Math, NMC Students Included 100 90 80 70 11 12 19 29 9 23 4 10 6 29 3 5 23 36 60 50 82 40 30 68 62 61 60 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 51 55 20 10 0 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Possible adjustments to Grade 7 cut • Currently the “meeting proficiency” cut is set on item 12 of the OIB, resulting in 85% of students classified as “developing proficiency. ” • In order to better articulate the impact data for Grade 7, IDOE recommended item 6 of the OIB be used for the Grade 7 “meeting proficiency” cut. Item 6 was identified during Round 2 of the process.

ISTAR Standard Setting Results – Math, Round 4, Grade 7 cut adjusted Impact Data for ISTAR Math after Round 4, Grade 7 adjusted (p. 6) 100 90 12 80 70 32 13 9 6 20 25 31 67 66 63 Grade 4 Grade 5 Grade 6 4 3 25 25 5 38 60 50 40 30 56 71 72 57 20 10 0 Grade 3 Developing Proficiency Meeting Proficiency Grade 7 Grade 8 Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – Math, all students, Grade 7 cut adjusted Impact data for ISTAR Math Grade 7, NMC Students Included; Grade 7 adjusted (p. 6) 100 90 80 70 11 12 9 6 4 3 23 23 5 19 23 29 62 61 60 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 29 36 60 50 40 30 51 69 68 55 20 10 0 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Mathematics: Context for Results • Grade 7 has been identified as being naturally harder than other grades in mathematics by the vertical articulation panel for the following reasons: • The PLDs, even at the “developing” level require more complex functioning of students than the PLDs from other grades (“understand” and “predict” vs. “identify”). • The content is different, calling for more abstract mathematical thinking (square roots and rational/irrational numbers). • Students are asked to use prior knowledge to apply their skills. They may or may not have this prior knowledge. • Grade 3 has been identified by the Vertical Articulation panel as possibly being easier for the students than other grades for, among others, the following reasons: • There are fewer components in Content Connectors. • Content is more concrete and applicable to everyday life and therefore more familiar to students.

Math Scale Score Cuts Meeting Proficiency Exceeding Proficiency Grade 3 353 379 Grade 4 365 386 Grade 5 354 381 Grade 6 348 379 Grade 7 347 380 Grade 8 350 382 Grade 10 352 390

Science

ISTAR Standard Setting Results – Science, Round 4 Impact Data for ISTAR Science after Round 4 100 90 17 31 80 50 70 60 38 50 41 40 31 30 20 45 28 10 19 0 Grade 4 Grade 6 Developing Proficiency Meeting Proficiency Grade 10 Exceeding Proficiency

ISTAR Standard Setting Results – Science, Vertical Articulation Impact Data for ISTAR Science after Vertical Articulation 100 90 17 31 80 41 70 60 42 50 41 40 40 30 20 41 28 10 19 0 Grade 4 Grade 6 Developing Proficiency Meeting Proficiency Grade 10 Exceeding Proficiency

ISTAR Standard Setting Results – Science, All Students Impact Data for ISTAR Science, NMC Students Included 100 90 16 30 80 39 70 60 39 38 50 38 40 30 38 20 27 19 10 0 7 5 4 Grade 6 Grade 10 No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Science Scale Score Cuts Meeting Proficiency Exceeding Proficiency Grade 4 354 388 Grade 6 355 393 Grade 10 342 383

Social Studies

ISTAR Standard Setting Results – Social Studies, Round 4 Impact Data for ISTAR Social Studies after Round 4 100 90 20 21 80 70 60 34 48 50 40 30 20 44 32 10 0 Grade 5 Developing Proficiency Grade 7 Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results – Social Studies, Vertical Articulation Impact Data for ISTAR Social Studies after Vertical Articulation 100 90 20 21 80 70 60 34 48 50 40 30 20 44 32 10 0 Grade 5 Developing Proficiency Grade 7 Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results – Social Studies, All Students Impact Data for ISTAR Social Studies, NMC Students Included 100 90 18 20 80 70 60 33 45 50 40 30 20 43 30 10 0 7 4 Grade 5 Grade 7 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Social Studies Scale Score Cuts Meeting Proficiency Exceeding Proficiency Grade 5 340 385 Grade 7 353 386