Learning about the Item Review Process An Overview

Learning about the Item Review Process: An Overview

Item Development Timeline • The first phase of item development contracts were awarded to ETS and Pearson in 2012. • Phase I of item development began in the fall of 2012 and was completed in late summer 2013. • Phase II began in October 2013 and will run through fall 2014. 2

Who are PARCC’s item reviewers? • PARCC item reviewers come from PARCC governing and participating states • K-16 educators, state department of education staff, and external experts • Deep content expertise • Experience with students from various backgrounds • Many have participated in item reviews for their own states • Experience in various geographic and educational settings 3

PARCC Item Review Committees Various review teams meet, both in person and virtually, to evaluate the items over an extended period of time. Only items that are approved by these teams of reviewers will appear on the PARCC summative assessments. 4 Group Membership Charge Core Leadership Groups PARCC state DOE staff and HE faculty; approx. 60 members in each content area Review all test items developed for the PARCC summative assessments for suitability of content, age-appropriateness, and alignment to CCSS. Approve recommended revisions to items. State Educator Reviewers K-12 LEA staff and HE faculty; approx. 80 members in each content area, plus 45 passage reviewers Ensure that items are age-appropriate and are measuring the content of the Common Core State Standards for a given grade level. Bias & Sensitivity Reviewers Citizens and educators Ensure that items and passages are fair, from various backgrounds; unbiased, and age-appropriate for a given grade approx. 50 members in level. each content area

Operational Working Groups • PARCC Operational Working Groups (OWGs) involved in item review: – ELA/Literacy – Mathematics – Accessibility, Accommodations, and Fairness (AAF) • OWG members are content experts from state departments of education • After each round of item review, OWGs participate in “item reconciliation” to review suggested revisions from the item review committees • Reconciliation following bias & sensitivity review includes members from content OWGs and AAF OWG 5

PARCC Item Review Process OWG Reconciliation Items Developed 6 Core Leadership Review OWG Reconciliation State Educator Review OWG Reconciliation Bias & Sensitivity Review Field Test Item Bank

PARCC Item Review Meetings • PARCC item review meetings are conducted both in-person and virtually, and last 3 – 5 days • Meetings consist of independent review of items followed by group discussion • ELA teams are organized by grade level (grades 9/10 are combined) • Math and Bias & Sensitivity teams are organized by grade band • Core Leadership groups have 4 -5 reviewers per team; State Educator and Bias & Sensitivity groups have 6 -8 reviewers per team 7

ELA Review Considerations/Criteria 1. Does the item allow for the student to demonstrate the intended evidence statement(s) and to demonstrate the CCSS to be measured? 2. Is the wording of the item clear, concise, and appropriate for the intended grade level? 3. Does the item provide sufficient information and direction for the student to respond completely? 4. Is the item free from internal clueing and miscues? 5. Do the graphics and stimuli included as part of the item accurately and appropriately represent the applicable content knowledge? 6. Are any graphics included as part of the item clear and appropriate for the intended grade level? 7. If the item has a technology-based stimulus or requires a technology-based response, is the technology design effective and grade appropriate? 8. Is the scoring guide/rubric clear, correct, and aligned with the expectations for performance that are expressed in the item or task? 9. If the item is part of a PBA task, does it contribute to the focus and coherence of the task model? 8

Mathematics Review Considerations/Criteria 9 1. Does the task measure the intended evidence statement(s)? 2. Does the task measure the intended mathematical practice(s)? 3. Is the task mathematically correct and free from errors? 4. Is the wording of the task clear, concise, and grade-level appropriate? 5. Are the graphics/stimuli in the task clear, accurate, appropriate for the task, and appropriate for the grade? 6. Do each prompt and all associated graphics/stimuli contribute to the quality of the task? 7. Is the scoring guide/rubric clear, correct and aligned with the expectations for performance that are expressed in the task?

Bias & Sensitivity Review Considerations/Criteria 1. Does the item disadvantage any population (gender, race, ethnicity, language, religion, socioeconomic status, disability or geographic region) for non-educationally relevant reasons? 2. Does the item contain controversial or emotionally charged subject matter that is not supported by the Common Core State Standards? 3. Is the item potentially offensive, demeaning, insensitive, or negative toward any population? 4. Does the item depict any population in a stereotypical 10 manner?

Questions?