ISTAR Standard Setting Overview of ISTAR Standard Setting

ISTAR Standard Setting

Overview of ISTAR Standard Setting Methodology • Item Descriptor (ID) Matching was suggested by the Indiana TAC for finding cut scores on the ISTAR tests. • ID matching process: • Evaluative process using an Ordered Item Booklet (OIB) and Performance Level Descriptors (PLDs) over several rounds • Based on the Performance Level Descriptors, panelists make judgment about each of the items in the OIB, classifying it as Developing, Meeting, or Exceeding proficiency (Rounds 1 and 2) • Panelists then indicate a “threshold region” where the item descriptors change from mostly lower PLD to mostly higher PLD (Rounds 1 and 2) • Within the threshold region, panelists are asked to find a transition point in the OIB where knowledge, skills, and cognitive processes required by items change from the lower PLD and begin to match more closely the knowledge, skills, and cognitive processes described in the higher level PLD (Rounds 3 and 4)

ISTAR Standard Setting Workflow • Meeting and Logistics: • General session — overview of ISTAR, introduction to the process, meeting logistics, housekeeping, security protocols • Experience the assessment — panelists utilize the testing environment mirroring the student experience and interaction with items • PLD discussion — panelists review and discuss the PLDs for the upper grade — what knowledge, skills, and cognitive processes expected of students classified in each performance level • OIB discussion — panelists review and discuss the OIB for the upper grade — what knowledge, skills, and cognitive processes are required to answer it correctly, and why is each item more difficult than the previous ones in the OIB • ID Matching practice round — panelists are assigned sample (10) items from the OIB to practice the tasks of matching items to PLDs

ISTAR Standard Setting Workflow (cont. ) • Meeting and Logistics (cont. ) • Rounds of judgments and feedback • 4 rounds of judgments (assign every item in the OIB to a PLD, identify threshold regions, and locate a cut score within each threshold region) • After each round, facilitator provides panelists with data that they can use to evaluate and inform their judgments in the subsequent rounds. Feedback data varied by round but generally included summary statistics (i. e. , descriptive statistics, such as the mean, median, minimum, and maximum page numbers) for the threshold region boundaries and recommended cut scores, agreement data, and impact data. • Repeat process with the lower grade-level assessment • Table Leaders continue to the Vertical Articulation meeting • Panelists examine the reasonableness of the proposed cut score recommendations across the grade levels

ISTAR Standard Setting Logistics • Standard Setting Meetings: • June 19– 21: ELA and Science • June 26– 28: Mathematics and Social Studies • Panels consisted of multiple breakouts: • Four grade band panels (3– 4, 5– 6, 7– 8 and 10) ELA and Mathematics • Two grade band panels (4 and 6, and 10) Science • One grade band panel (5 and 7) Social Studies • Each grade band panel had a facilitator and a content specialist • Each panel consisted of 8– 9 Indiana Educators in two tables; most panels had one general education teacher • Two of the panelists were designated by IDOE as table leaders — they led table discussions, took notes, and spoke on behalf of the table during the large-group discussion

ISTAR Performance Levels • Two cuts were recommended for each grade and subject: • The Meeting Proficiency cut — The cut score that differentiates Developing Proficiency performance from Meeting Proficiency performance. • The Exceeding Proficiency cut — The cut score that differentiates Meeting Proficiency performance from Exceeding Proficiency performance.

ISTAR Standard Setting Results • Tables and graphs in the coming slides present the impacts of cut scores • Three impacts are given for all grades (except Grade 7 Mathematics) • Impacts of cuts recommended in Round 4 by grade band panels • Impacts of cuts adjusted in vertical articulation committees, if adjustments were made • Impact of IDOE recommendation for Grade 7 Mathematics • Impacts of cuts on the whole population which includes the No Mode of Communication students reported as “Undetermined” proficiency level

ISTAR Standard Setting Results – ELA, Round 4 Impact Data for ISTAR ELA after Round 4 100 90 26 41 80 42 34 38 44 70 60 53 34 50 35 40 39 48 44 36 34 30 20 39 10 23 19 18 18 19 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 14 0 Grade 3 Developing Proficiency Meeting Proficiency Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – ELA, Vertical Articulation Impact Data for ISTAR ELA Vertical Articulation 100 90 26 80 41 42 35 39 34 38 48 44 44 46 70 60 50 40 40 36 30 20 34 10 23 19 18 18 19 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 40 14 0 Grade 3 Developing Proficiency Meeting Proficiency Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – ELA, all students Impact Data for ISTAR ELA, NMC Students Included 100 90 24 80 32 36 45 42 34 18 18 39 39 32 44 70 60 36 50 40 30 20 10 0 31 22 39 9 7 7 5 4 6 13 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ELA: Context for Results • The complexity of passages may not be as high as expected. Development of passage expectations to ensure distribution of complexity is ongoing. • Using supporting graphics in some items may make them less difficult. • Specifically, the Vertical Articulation panel had the following thoughts about grade 3 impact data: • PLDs were seen as being more difficult in this grade than in other grades • Grade 3 is the first year the students take the assessment, so they might be struggling with the concept of test-taking • Revisit the assessment of writing in terms of the overall blueprint and relationship to the general education assessment

ISTAR Standard Setting Results – Mathematics, Round 4 Impact Data for ISTAR Math after Round 4 100 90 12 80 70 32 13 20 9 25 6 3 4 11 31 5 25 38 60 50 85 40 30 56 67 66 63 Grade 4 Grade 5 Grade 6 72 57 20 10 0 Grade 3 Developing Proficiency Meeting Proficiency Grade 7 Grade 8 Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – Mathematics, all students Impact Data for ISTAR Math, NMC Students Included 100 90 80 70 11 12 19 29 9 23 4 6 10 29 3 5 23 36 60 50 40 30 82 68 51 62 61 60 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 55 20 10 0 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results – Mathematics Grade 7 IDOE Recommendation Impact Data for ISTAR Math after Round 4 100 90 12 80 70 32 13 9 6 20 25 31 67 66 63 Grade 4 Grade 5 Grade 6 4 3 25 25 5 38 60 50 40 30 56 71 72 57 20 10 0 Grade 3 Developing Proficiency Meeting Proficiency Grade 7 Grade 8 Exceeding Proficiency Grade 10

ISTAR Standard Setting Results – Mathematics Grade 7 IDOE Recommendation Impact data for ISTAR Math Grade 7, NMC Students Included; Grade 7 adjusted (p. 6) 100 90 80 70 11 12 9 6 4 3 23 23 5 19 23 29 62 61 60 9 7 7 5 4 6 4 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 29 36 60 50 40 30 51 69 68 55 20 10 0 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

Mathematics: Context for Results • Continued focus on rigor given the transition to the new standards • Middle school grades have been identified as being naturally harder than other grades in mathematics by the vertical articulation panel for the following reasons: • The PLDs, even at the “developing” level require more complex functioning of students than the PLDs from other grades (“understand” and “predict” vs. “identify”). • The content is different, calling for more abstract mathematical thinking (square roots and rational/irrational numbers). • Students are asked to use prior knowledge to apply their skills. They may or may not have this prior knowledge. • Grade 3 has been identified by the Vertical Articulation panel as possibly being easier for the students than other grades for, among others, the following reasons: • There are fewer components in Content Connectors. • Content is more concrete and applicable to everyday life and therefore more familiar to students.

ISTAR Standard Setting Results – Science, Round 4 Impact Data for ISTAR Science after Round 4 100 90 17 31 80 50 70 60 38 50 41 40 31 30 20 45 28 10 19 0 Grade 4 Grade 6 Developing Proficiency Meeting Proficiency Grade 10 Exceeding Proficiency

ISTAR Standard Setting Results – Science, Vertical Articulation Impact Data for ISTAR Science Vertical Articulation 100 90 17 31 80 41 70 60 42 41 50 40 40 30 20 41 28 10 19 0 Grade 4 Grade 6 Developing Proficiency Meeting Proficiency Grade 10 Exceeding Proficiency

ISTAR Standard Setting Results – Science, all students Impact Data for ISTAR Science, NMC Students Included 100 90 16 30 80 70 60 39 39 38 50 38 40 30 38 27 20 19 10 0 7 5 4 Grade 6 Grade 10 No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results – Social Studies, Round 4 Impact Data for ISTAR Social Studies after Round 4 100 90 20 21 80 70 60 34 48 50 40 30 20 44 32 10 0 Grade 5 Developing Proficiency Grade 7 Meeting Proficiency Exceeding Proficiency

ISTAR Standard Setting Results – Social Studies, all students Impact Data for ISTAR Social Studies, NMC Students Included 100 90 18 20 80 70 60 33 45 50 40 30 20 43 30 10 0 7 4 Grade 5 Grade 7 Percentage of Students No Mode of Communication Developing Proficiency Meeting Proficiency Exceeding Proficiency