MCAS 2019 Preliminary Data and Discrepancy Reporting Overview

MCAS 2019 Preliminary Data and Discrepancy Reporting Overview July 2019

Presenters • Bob Lee, MCAS Chief Data Analyst • Scott Kelley, Reporting Specialist • Jodie Zalk, Test Administration Coordinator Massachusetts Department of Elementary and Secondary Education 2

Logistics for This Session • Due to this year’s early discrepancy reporting period, this session is only available as a recording, and is posted at: o MCAS Resource Center -- mcas. pearsonsupport. com/training o In the section entitled “Training Sessions Previously Offered This School Year” • Questions should be directed to the appropriate group, as follows: o Questions about this session or testing policy should be directed to Student Assessment at mcas@doe. mass. edu or 781 -338 -3625. o Questions about the Discrepancy Reporting Tool should be directed to the MCAS Service Center at mcas@measuredprogress. org or 800737 -5103. Massachusetts Department of Elementary and Secondary Education 3

Data Deliveries and Early 2019 01 MCAS Results 02 What’s New CONTENTS 03 Discrepancy Reporting Window

01 Data Deliveries and Early 2019 MCAS Results

2019 Data Delivery Milestones • April 29: Grade 10 ELA machine scored results. • June 11: MCAS-Alt Portfolio Feedback Forms in Drop. Boxes • June 18: ELA Grade 10 data in Pearson. Accessnext (no scaled scores or SGP; released essays on June 24) • July 1: Grades 3– 8 ELA and Math; Grades 5 & 8 STE (ELA released essays; no scaled scores or SGP) • July 8: Grade 10 Math (no scaled scores or SGP) • July 24 in Drop. Boxes: Raw scores for grades & subjects (. csv only; no rosters, scaled scores, or SGP) • July 29 in Drop. Boxes: Preliminary participation data (aggregated by subgroup & subject) Massachusetts Department of Elementary and Secondary Education 6

2019 Data Delivery Milestones (cont’d) • July 31 in Edwin Analytics: Raw scores for all grades and subjects (curriculum & item analysis interactive reports; no scaled scores or SGP) • August 23 in Drop. Boxes: Full preliminary results for all grades and subjects (includes scaled scores, achievement levels, SGP; . csv data files only; no rosters) • August 30 in Edwin Analytics: Full preliminary results for all grades and subjects (includes scaled scores, achievement levels, SGP) • Mid-September: Official embargoed MCAS rosters and. csv files in Drop. Boxes; interactive reports in Edwin Analytics • Late September: Public release of official MCAS results • Official state, district and school-level results on profiles. doe. mass. edu • Parent/Guardian reports received by superintendents Massachusetts Department of Elementary and Secondary Education 7

Analyzing Early Release Raw Data • Excel Template: Statewide item information, state average points and percent correct per item (item difficulty) available for next-generation MCAS tests at: § Pearson. Accessnext in Support>Documentation § MCAS Resource Center in Released Items (See link below. ) • Compare school & district points/percent correct to state averages. • Instructions for using the templates are available in both places. • Released items for next-generation tests are available on the MCAS Resource Center. § http: //mcas. pearsonsupport. com/released-items/ § Includes all items and item descriptions for ELA, Math, and STE operational items Massachusetts Department of Elementary and Secondary Education 8

Using the item analysis template (from PAN / Resource Center) Massachusetts Department of Elementary and Secondary Education 9

Using the item analysis template: pre-formatted graph Massachusetts Department of Elementary and Secondary Education 10

02 What’s New

Next-Generation Grade 10 ELA Test Design vs. Legacy Next-Generation Grade 10 ELA Test Design Legacy Grade 10 ELA Test Design Item Type Number of questions Points Possible Composition Points Breakdown Topic Develop ment Composition Conventions Percent of Total Points One-Point Multiple Choice 36 36 n/a 50% Four-Point Open Response 4 16 n/a 22% Composition Total 1 41 20 72 12 8 28% 100% Item Type One-Point Selected Response Two-Point Multiple Select/Tech. Enhanced Essay Total Number Points of Possible questions Essay Points Breakdown Idea Development Standard English Conventions 21 -27 n/a 41%– 53% 4 -7 8 -14 n/a 16%– 27% 2 31 16 51 10 6 31% 100% ELA Test Designs www. doe. mass. edu/mcas/tdd/ela. html? section=testdesign Massachusetts Department of Elementary and Secondary Education Percent of Total Points 12

Next-Generation Grade 10 Math Test Design vs. Legacy Grade 10 Math Test Design Next-Generation Grade 10 Math Test Design Item Type One-Point Multiple Choice Number of questions 32 Points Possible 32 Percent of Total Points 53% One-Point Short Answer 4 4 7% Four-Point Open Response 6 24 40% Total 42 60 100% Item Type Number of questions Points Possible Percent of Total Points One-Point Selected Response 32 32 53% Two-Point Multiple Select/Tech. Enhanced 6 12 20% Four-Point Constructed Response 4 16 27% Total 42 60 100% Math Test Designs www. doe. mass. edu/mcas/tdd/math. html? section=testdesign Massachusetts Department of Elementary and Secondary Education 13

Next-Generation Grades 5 & 8 STE Test Design vs. Legacy Grades 5 & 8 STE Test Design Next-Generation Grades 5 & 8 STE Test Design Item Type One-Point Multiple Choice Four-Point Open Response Total Number of questions 38 Points Possible 38 Item Type Number of questions Points Possible 32 32 70% One-Point Selected Response/Tech. Enhanced Percent of Total Points 60% Two-Point Multiple Select/Tech. Enhanced 3 6 11% Two-Point Constructed Response 2 4 7% Three-Point Constructed Response Total 4 12 22% 41 54 100% Percent of Total Points 4 16 30% 42 54 100% STE Test Designs www. doe. mass. edu/mcas/tdd/sci. html? section=testdesign Massachusetts Department of Elementary and Secondary Education 14

2019 vs. 2018 ELA Average Raw Score Stats ELA Change in Percent Correct 5% 1% 0% -1% -5% -2% -1% -2% -4% -6% -10% -15% -20% ELA 03 ELA 04 ELA 05 ELA 06 Massachusetts Department of Elementary and Secondary Education ELA 07 ELA 08 * ELA 10 Test/ Points Grade Possible ELA 03 44 ELA 04 44 ELA 05 48 ELA 06 51 ELA 07 51 ELA 08 51 ELA 10 51 / 72 2019 Mean Raw Score 25. 1 26. 0 28. 2 27. 4 28. 1 29. 0 37. 4 2018 Mean Raw Score 25. 5 27. 9 29. 1 28. 1 31. 4 30. 1 52. 4 Changes in raw scores do not necessarily mean state results will go down. *Transition to Next-Gen MCAS in 2019 15

2019 vs. 2018 Math Average Raw Score Stats Math Change in Percent Correct 5% 3% 1% 1% 2% 1% 0% -1% -5% -10% -9% -15% -20% MAT 03 MAT 04 MAT 05 MAT 06 Massachusetts Department of Elementary and Secondary Education MAT 07 MAT 08 * 2019 Mean Test/ Points Raw Grade Possible Score MAT 03 48 28. 0 MAT 04 54 33. 0 MAT 05 54 29. 5 MAT 06 54 29. 9 MAT 07 54 25. 2 MAT 08 54 29. 7 MAT 10 60 34. 4 2018 Mean Raw Score 27. 5 31. 3 28. 8 28. 6 24. 6 30. 2 40. 0 Changes in raw scores do not necessarily mean state results will go down. MAT 10 *Transition to Next-Gen MCAS in 2019 16

2019 vs. 2018 STE Average Raw Score Stats STE Change in Percent Correct 20% 18% Test/ Grade 16% 2019 Mean Points Raw Possible Score 2018 Mean Raw Score 14% STE 05 54 31. 2 26. 50 12% STE 08 54 28. 9 26. 15 10% 9% 8% 5% 6% 4% 2% 0% SCI 05 * Massachusetts Department of Elementary and Secondary Education SCI 08 * *Transition to Next-Gen MCAS in 2019 17

CBT Released ELA Essays in PDF for All Grades ELA released essay responses for ELA grades 3‒ 8 & 10 in Pearson. Accessnext § Available in school’s Published Reports § One file per grade § Released items posted at mcas. pearsonsupport. com/releaseditems/ela Massachusetts Department of Elementary and Secondary Education 18

Superintendent Survey on Parent/Guardian Report Preference • DESE surveyed superintendents in late February 2019 regarding districts’ preference in receiving parent reports in paper and/or electronic form. • Sixty-eight responses were received to the following three questions. Massachusetts Department of Elementary and Secondary Education 19

Superintendent Survey on Parent/Guardian Report Preference Massachusetts Department of Elementary and Secondary Education 20

Superintendent Survey on Parent/Guardian Report Preference • Representative comments from the survey include: § Hard copy goes in student cumulative folder. Until electronic cumulative folders are widespread, two hard copies will have to be provided. Sending parents the pdf would be cumbersome as matching student report to parent email would require significant manual labor in school. § My concern with not receiving any paper copies is that I think parents can/should receive a personalized report and I don't think the printing burden should be shifted to districts. We could make do with one paper copy to be mailed home and just use electronic copies for internal purposes. § The electronic version would be very challenging in small rural areas where some families still do not utilize internet. Paper copies are much appreciated. § The biggest issue is distribution of reports to multiple households when a child's parents are divorced. Massachusetts Department of Elementary and Secondary Education 21

Grade 10 ELA Essay Item Not Scored • ELA Grade 10 essay item 27 was not scored. Results for the item will be reported on the Parent/Guardian Report as follows: Massachusetts Department of Elementary and Secondary Education 22

Grade 10 Next-Gen Parent/Guardian Report • Scaled score, SGP, and Competency Determination status are on p. 1 • Inside and back pages are similar to grades 3– 8 reports. Massachusetts Department of Elementary and Secondary Education 23

Equipercentile linking model for establishing the interim CD standards for the Classes of 2021 & 2022 (ELA example) 100 90 80 70 51% Adv 60 50 % ~260 For Adams Scholarship 40 30 20 10 0 40% Prof 6% NI ~240 For CD without EPP ~220 Interim CD Standard 3% F MCAS 2018 Grade 10 Level 1 Massachusetts Department of Elementary and Secondary Education Next Gen Gr 10 (simulated) Level 2 Level 3 Level 4 24

Estimated Equipercentiles for Grade 10 ELA and Math Legacy MCAS ELA 2018 Raw Score: Scaled Score 21 -22: 220 36 -37: 240 55 -56: 260 Percentile 3 rd 9 -10 th 51 -55 th Legacy MCAS Math 2018 Raw Score: Scaled Score 17: 220 28: 240 42 -44: 260 Percentile 8 th 22 nd 50 -54 th Massachusetts Department of Elementary and Secondary Education next-generation MCAS ELA 2019 Estimated Raw Score 12 -13 21 -22 40 -41 % correct (of 51 points) 24 -25% 41 -43% 78 -80% next-generation MCAS Math 2019 Estimated Raw Score 11 -12 20 35 -38 % correct (of 60 points) 18 -20% 33% 58 -63% 25

Next-Gen 2019 MCAS Released Item Sets • The Department posted interactive released item sets in Test. Nav on the MCAS Resource Center at mcas. pearsonsupport. com/released-items. o works in a browser; provides selected response score • PBT Released item sets are posted on the Department’s MCAS Test Questions page at www. doe. mass. edu/mcas/release. html. o includes high school Science and Technology/ Engineering Massachusetts Department of Elementary and Secondary Education 26

Changes Between Previous Versions and the New Data File • Released typed response essays will be available in PAN • Medical absences, First-Year EL exemptions noted • Security breaches invalidated (cell phone use, inappropriate accommodations, cheating cases that have been resolved) • Duplicate cases resolved, small numbers of data processing errors corrected Massachusetts Department of Elementary and Secondary Education 27

Using Preliminary Results – Embargo Policies Do • Use preliminary results for educational planning purposes and research with appropriate staff. Don’t • Discuss aggregate results in public. • Show preliminary aggregate reports at school committee meetings. • You may share preliminary results with students and families, but take care not to imply certainty about preliminary results. Massachusetts Department of Elementary and Secondary Education 28

03 Discrepancy Reporting Window

MCAS Discrepancy Reporting • July 24−August 6, 5 p. m. • CSV data files are now in Drop. Boxes • Discrepancy reporting tool is at www. mcasservicecenter. com o Use IE 8− 10; Firefox 17+; Safari 5− 6 o Do not use Chrome • MCAS Service Center passwords were distributed in January 2019 • Spring 2019 MCAS Guide to Reporting a Discrepancy is posted both places • Call 800 -737 -5103 if you need assistance Massachusetts Department of Elementary and Secondary Education 30

How to Review Your Data for Potential Discrepancies • Review individual student data using data files (now in Drop. Box; July 31 in Edwin Analytics) • Confirm that each student has a SASID. • Confirm that each student enrolled during the testing windows is listed only once. • Verify that each student’s test participation status is correct (the Discrepancy Reporting Guide contains explanations of the codes). Massachusetts Department of Elementary and Secondary Education 31

Test Participation and Test Status Code used on Disc. Rept. tool ABS ALT ALN ALR DNT DUP INC INV LEP MED PAS PPR PRF REN RET STD TRN Status Absent Tested Alternate Assessment Retested Alternate Assessment Not Aggregated Retested Alternate Assessment Did Not Test Multiple Answer Booklets Incomplete Invalidated First-year LEP Absent - Medically Documented Previously Passed and Retested Previously Failed Retested Not Aggregated Retest Student Tested Standard Transferred . csv & reporting Codes ABS T Description Not tested or partially tested Tested MCAS-Alt TR Grade 10 MACS-Alt retest with prior grade 10 attempt TR Grade 10 MCAS-Alt STE attempt, with previously failed grade 9 attempt NTO NTO NTL NTM Not tested & not required to test More than one booklet received for same student Partial test not scored; repeated CBT tech issues; no accountability reporting Invalidated due to security breach First-year EL: satisfied participation requirement; scaled score and SGP not reported Did not test for medical reasons NTO Did not test: earned a passing score on a prior high school STE test Did not test: earned a passing score on a prior high school STE retest NTO TR TR T NTO Did not test: did not earn a passing score on a prior high school STE test Grade 10 attempt with an existing prior grade 10 attempt Grade 10 STE attempt with an existing prior grade 9 STE attempt Fully tested and reported for accountability Transferred into our out of your school/district; not tested and not reported for accountability Voided Answer Booklet Massachusetts Department of and. Answer Secondary Education VAB Elementary Voided Booklet NTO 32

Discrepancies Schools/Districts Can Change Directly • A missing SASID can be added. • Test participation status can be changed from “Medically documented” to “Absent” or vice-versa. • Incorrectly voided tests can be un-voided (all other changes to participation status must be reviewed by Department staff). Massachusetts Department of Elementary and Secondary Education 33

Discrepancies Requiring DESE Data Review • Student was not tested at your school (or an affiliated test site) but appears on your roster with test results • Incorrect or invalid SASID can be corrected; • Test results belong to a different student • Student was fully tested but does not appear in results • Student has two sets of results (listed as “DUP”) • Student is reported as “Absent” but was not enrolled at your school during testing window • Student is reported as “Absent” but was tested and received a response accommodation (typed or scribed response) • Participation status should be changed from “Absent” to “Tested” (student confirmed as present during all test days) Massachusetts Department of Elementary and Secondary Education 34

Discrepancies Requiring DESE Data Review (continued) • Participation status should be changed to “First-year EL” or “Transferred” (subject to Department validation using SIMS) • Special access accommodation status is incorrect • High school only: o Student is reported as “Absent” for ELA and/or Math tests but previously participated in a spring grade 10 test o summarize = 1 but student was not required to test o CD Status is not yet available o A grade 10 student is reported with a test status of “TR” (Tested–Retest) Massachusetts Department of Elementary and Secondary Education 35

What Not to Report as a Discrepancy • Students with correct SASID but incorrect name, birth date, gender, race/ethnicity, or any other SIMS data; update next SIMS submission • Students involved in a test breach whose results appear as “INV. ” These results have been invalidated due to a security breach. • Correctly voided CBT test or answer booklets (listed as “VAB”) • Students reported as enrolled in March and June 2019 SIMS, did not participate in test(s), and therefore assigned a performance level of “Absent”: update enrollment in next SIMS submission • Due to the proven accuracy of paper test-booklet scanning, missing items and groups of missing selected response items should not be reported for paper-based test-takers Massachusetts Department of Elementary and Secondary Education 36

What Not to Report as a Discrepancy (continued) • Classroom reports—incorrect Student Course Schedule data • High school only: o Student who skipped grade 10 and did not participate in grade 10 spring ELA and Math tests and is reported as “Absent” (retests do not count) Massachusetts Department of Elementary and Secondary Education 37

Email and Phone Support MCAS Service Center • All questions on logistics and technology (e. g. , Pearson. Access Next, SR/PNP, Test. Nav) o Web: mcas. pearsonsupport. com/ o Email: mcas@measuredprogress. org o Phone: 800 -737 -5103 Massachusetts Department of Elementary and Secondary Education DESE Office of Student Assessment Services • All policy questions (e. g. , student participation, accommodations) o Web: www. doe. mass. edu/mcas o Email: mcas@doe. mass. edu o Phone: 781 -338 -3625 38

THANK YOU The Office of Student Assessment Services 781 -338 -3625 www. doe. mass. edu/mcas@doe. mass. edu 75 Pleasant Street, Malden, MA 02148