Establishing MME and MEAP Cut Scores Consistent with

Establishing MME and MEAP Cut Scores Consistent with College and Career Readiness A study conducted by the Michigan Department of Education (MDE) and ACT, Inc.

Overview of MEAP and MME n n MEAP (Michigan Educational Assessment Program) MME (Michigan Merit Examination)

Why Reset Cut Scores Now? n Disjoint cut scores Relative difficulty of cut scores increases by grade in grades 3 -8 ¨ Relative difficulty spikes in high school (grade 11) ¨ n Relatively low cut scores ¨ n Based on standard setting panelists’ interpretation of “basic skills” New economy Requires a redefinition of “basic skills” ¨ Need students to be prepared for technical career training or college ¨ n State Board of Education History Adopted new, rigorous high school content standards in 2006 ¨ Adopted new high school exam in 2006 ¨ Did not want to adopt college ready cut scores at that time (wanted to give time for schools to implement the new content standards) ¨ Requested that in five years time the issue be revisited to raise cut scores ¨

Previous Impact Data 100 95 90 Percent Proficient 80 87 91 84 85 80 70 84 84 85 79 82 78 60 63 50 52 40 Reading Math 30 20 10 0 3 4 5 6 Grade Level 7 8 11

Purpose of the study n Set new cut scores representing being on track to career and college readiness in the grades and subjects in red n Not needed in writing, since the writing standards were originally set to be consistent with ACT’s college readiness benchmarks Need three cut scores to create four performance levels n Advanced ¨ Proficient ¨ Partially Proficient ¨ Not Proficient (no cut score needed) ¨

Standard Setting n Reasoned Judgment Subjective (panelist judgment) ¨ With objective elements (e. g. , ordered item booklets) ¨ n Empirical ¨ Objective n ¨ With subjective elements n n External criterion (grade in related freshman course) was not created for the purpose of standard setting External criterion (freshman college course grade) is itself a subjective judgment on the part of the college professor Choice Empirical ¨ Meaning ¨

Two types of linkages were needed to identify the cut scores: n n Link 11 th grade MME scores to freshman college grades to identify cut scores on the MME. Link MEAP scores to MME scores to identify cut scores on one or more grades of the MEAP.

Step 1: The MME n To do list ¨ Define what is meant by college success ¨ Collect data on students with MME scores and college outcomes ¨ Decide on a method that links MME scores and college success

Measure of college success First year courses n Course grade- either average grade, or specific grade or better n Examples: ¨ average grade of 3. 0 ¨ Proportion of students with a grade of B or better

The MME - Data n MME Test data available with MI student ID n CEPI data available with MI student ID This is a very good thing

The MME data College grade data taken from Fall 2009 and Fall 2010 n Course grade data from 39 public colleges n 19 4 -year institutions ¨ 20 2 -year institutions ¨ n Each college was given the opportunity to define their “first year courses” in each course area

The MME - data n Number of courses varied by subject Mathematics 1 course College Algebra Science 4 courses Biology/Life Science General Chemistry Physics Other

The MME – College Grade Data n Total sample sizes: ¨ Reading ¨ Mathematics ¨ Science ¨ Social Science 40, 164 6, 567 16, 180 41, 780 >100, 000 records

The MME - Methods Signal Detection Theory (SDT) Very n Logistic Regression (LR) Similar n Conditional mean n

The SDT Method Not Successful/ Successful Cut score (To be determined) Not Successful/ Not Successful College grade

The Logistic Method 0. 5 Cut score

The MME n To get a cutscore, we must choose a success criterion and a method ¨B or higher grade ¨ SDT method

MME – the results Subject Proficient Cut Score 1116 Past Proficient Cut Score 1100 Reading 1108 1100 Science 1126 1100 Social Science 1129 1100 Mathematics

Step 2: MEAP to MME What does it mean to say that a 5 th grade student is proficient? General answer: A student is CCR if his/her score in grade 5 indicates that the student is on track to be college ready in grade 11

College Readiness in Grade 5 Not College Ready

What link to make? n Option 1 – Link Grade 8 to Grade 11, Grade 7 to Grade 8, etc. Advantage: Larger Sample Size n Option 2 – Link Grade 8 to Grade 11, Grade 7 to Grade 11, etc Advantage: Less Measurement Error

Cohort with at Least Two Years of Data Grade Sample Size Cohort N_ Totalb N_Full_ Matchedc 3 4 5 6 7 8 9 10 a 11 12 1 09 -10 10 -11 - - - - 118, 468 107, 844 2 08 -09 09 -10 10 -11 - - - - 123, 896 105, 019 3 07 -08 08 -09 09 -10 10 -11 - - - 128, 557 101, 277 4 06 -07 07 -08 08 -09 09 -10 10 -11 - - - 135, 268 98, 131 5 05 -06 06 -07 07 -08 08 -09 09 -10 10 -11 - - 140, 729 94, 044 6 - 05 -06 06 -07 07 -08 08 -09 09 -10 10 -11 - - - 149, 950 93, 447 7 - - 05 -06 06 -07 07 -08 08 -09 09 -10 10 -11 - - 151, 047 98, 155 8 - - - 05 -06 06 -07 07 -08 08 -09 09 -10 10 -11 - 161, 869 88, 526 9 - - 05 -06 06 -07 07 -08 08 -09 09 -10 10 -11 163, 246 93, 004 10 - - - 05 -06 06 -07 07 -08 08 -09 09 -10 160, 813 97, 172 11 - - - 05 -06 06 -07 07 -08 08 -09 158, 255 104, 352

Fewer links – smaller(? ) sample MEAP scores Grade 3 Grade 4 Grade 5 Grade 6 MEAP scores Grade 7 Grade 8 Grade 9 MME scores Grade 11

MEAP – the results (mathematics) Grade New Cut Score Old Cut Score 8 830 800 7 731 700 6 629 600 5 531 500 4 434 400 3 338 300

Validity checks n Consistency of classification across years ¨ Should n be high, but not too high Consistency of classification with different tests (like the ACT) ¨ Note that MME contains items from the ACT as one of its constituent parts for Reading, Math, and Science

Classification Consistency Rates Year to Year Grade 11 -grade Mathematics Partially Proficient - Proficient 65% Advanced - 8 -11 83% 86% 95% 7 -8 81% 84% 95% 6 -7 82% 83% 96% 5 -6 81% 84% 95% 4 -5 80% 82% 94% 3 -4 77% 80% 95%

Consistency of MME with ACT Math Meeting College and Career Readiness Benchmarks MME Math Proficient Yes No 30, 534 25. 5% 7, 004 5. 8% 1, 241 1. 0% 81, 041 67. 6% 93%

Consistency of MME with ACT Subject More difficult Mathematics Consistency Rate 93% Reading 86% ACT Science 90% ACT MME

Consequences – Reading (Student Level)

Consequences – Reading (Elementary and Middle School Level)

Consequences – Reading (High School Level)

Consequences – Mathematics (Student Level)

Consequences – Mathematics (Elementary and Middle School Level)

Consequences – Mathematics (High School Level)

Message Michigan schools are not adequately preparing students for freshman-level college/career-training courses n Michigan schools are better preparing students for entry-level reading-heavy (humanities, social science) courses than for entry-level mathematics courses n

Message n n Florida recently attempted to do this in writing, but the increase in cut scores was rejected We believe Michigan was able to successfully make the transition to more rigorous cut scores because of advance notice to the field, the press, and the state board of education regarding the anticipated impact and a strong argument for the need to revisit cut scores

Thank you for your attention Any Questions or Comments?