Refinement Year Results 2016 2017 1192018 9102021 Study

Refinement Year Results (2016 -2017) – 1/19/2018 9/10/2021

Study Participation and Representativeness These slides summarize results from the Refinement Year of the Multi-State Collaborative to Advance Quality Student Learning involving 13 states using common rubrics to assess more than 7, 000 pieces of student work. 9/10/2021

Participating States 13 States Connecticut, Hawaii, Indiana, Kentucky, Maine, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Texas, Utah, and Virginia 9/10/2021

7, 386 papers were submitted All students were near graduation. By the time that students graduate, are they proficient in writing, presenting, and interpreting data? Are they proficient at thinking critically? Critical Thinking Quantitative Literacy Written Communication Total Associates 1, 131 449 982 2, 562 Bachelors 2, 024 782 2, 018 4, 824 Total 3, 155 1, 231 3, 000 7, 386 Note: Excludes “early” students, students with an unknown degree level, duplicates, and students with credits outside of the MSC project range. 9/10/2021

Representativeness: Gender MSC Project Sample Demographics Relative to Graduating Students at Participating Institutions Associates 62% Bachelors 61% MSC IPEDS Female MSC Male IPEDS 45% 39% 38% IPEDS 58% 55% IPEDS MSC Female MSC Comparison data used are 2015 IPEDS graduates (associates, bachelors) at participating institutions. 9/10/2021 IPEDS 42% MSC Male IPEDS MSC

Representativeness: Age MSC Project Sample Demographics Relative to Graduating Students at Participating Institutions Associates 24 and Under 100% 15% 25 -39 Bachelors 40+ 24 and Under 8% 28% 75% 40% 100% 75% 25 -39 40+ 5% 6% 23% 20% 72% 74% IPEDS MSC 50% 25% 45% 64% 25% 0% 0% IPEDS MSC Comparison data used are 2015 IPEDS graduates (associates, bachelors) at participating institutions. 9/10/2021

Representativeness: Race/Ethnicity MSC Project Sample Demographics Relative to Graduating Students at Participating Institutions AI/PI/NA Associates IPEDS 67% White MSC 71% White 20% 18% Asian Black or African American 16% Hispanic or Latino 14% 12% Non-resident alien 12% 10% 9% 9% Two or more races 10% Unknown 8% 6% 4% 2% 4% 3% 1% 1% 3% 0% MSC IPEDS Comparison data used are 2015 IPEDS graduates (associates, bachelors) at participating institutions. 9/10/2021 3%

Representativeness: Race/Ethnicity MSC Project Sample Demographics Relative to Graduating Students at Participating Institutions 20% 18% Bachelors AI/PI/NA IPEDS 68% White MSC 68% White Asian Black or African American 16% Hispanic or Latino 14% Non-resident alien 12% 10% Two or more races 8% 6% 5% 6% 3% 4% 2% 8% 8% 2% 1% Unknown 7% 3% 3% 4% 4% 1% 0% MSC IPEDS Comparison data used are 2015 IPEDS graduates (associates, bachelors) at participating institutions. 9/10/2021

Massachusetts 2016 -17 Distribution of Student Scores

Massachusetts Participating Institutions: 2016 and 2017 Institution 2016 Participants 2017 Participants Bristol Community College No Yes Massasoit Community College No Yes Mount Wachusett Community College Yes Northern Essex Community College Yes North Shore Community College No Yes Quinsigamond Community College Yes No Fitchburg State University No Yes Salem State University Yes Westfield State University Yes Worcester State University No Yes Univ. of Massachusetts Lowell Yes Univ. of Massachusetts Amherst Yes TOTAL 7 11

Multi State Collaborative Sample Sizes 2015 -2016 -2017 Net Change (%) Critical Thinking 840 1, 283 443 (52. 7%) Written Communication 919 990 71 ( 7. 7%) Quantitative Literacy 576 381 195 (-33. 9%) Total – 2 yr colleges 2, 335 2, 654 319 (13. 7%) Critical Thinking 2, 056 2, 006 50 ( -2. 4%) Written Communication 1, 936 2, 123 187 ( 9. 7%) Quantitative Literacy 787 748 39 ( -5. 0%) Total – 4 yr colleges 4, 779 4, 877 98 ( 2. 1%) Critical Thinking 108 312 204 (189%) Written Communication 123 281 158 (128%) Quantitative Literacy 24 38 Total – 2 yr colleges 255 631 376 (147%) Critical Thinking 378 484 106 ( 28. 0%) Written Communication 259 372 113 ( 43. 6%) Quantitative Literacy 43 111 68 (158%) Total – 4 yr colleges 680 967 287 (42. 2%) Project Level 2 yr colleges 4 yr colleges Massachusetts Level 2 yr colleges 4 yr colleges 14 ( 58. 3%)

Massachusetts Critical Thinking Distribution - 2017 2 -Year Institutions Explanation of Issues Milestone 4 5% 4% Milestone 3 28% Milestone 2 Zero 0 17% 50% 17% 4 -Year Institutions Milestone 4 Milestone 3 Benchmark 1 Zero 0 40% 42% 11% 2% 20% 3% Conclusion 10% 38% 28% 33% 39% 9% 6% Student's Position 29% 38% 30% 11% 6% 9% 25% 37% 11% Context and Assumptions Evidence 7% 15% Milestone 2 10% 50% 35% 33% 10% Explanation of Issues 10% 40% 19% 8% 4% 14% Conclusion Student's Position 3% 3% 42% Benchmark 1 Context and Assumptions Evidence 19% 4% 44% 14% 4%

Massachusetts Written Communication Distribution - 2017 2 -Year Institutions Context/Purpose Milestone 4 Milestone 3 4 -Year Institutions Zero 0 6% 9% 1% 0% Milestone 2 Benchmark 1 Zero 0 0% 1% 23% Genre and Conventions 5% 18% 21% Syntax and Mechanics 0% 6% 33% 44% 2% 52% Sources and Evidence 27% 14% 26% 13% 38% 33% 15% 32% 39% 4% 35% 2% 8% 13% Milestone 3 23% 47% Content Development Syntax and Mechanics 0% 15% 30% 22% Sources and Evidence 5% 36% 44% Context/Purpose Milestone 4 Genre and Conventions 4% 27% 23% Milestone 2 Benchmark 1 Content Development 8% 41% 34% 13% 14% 41% 9%

2 -Year Institutions Massachusetts Quantitative Literacy Distribution - 2017 Interpretation Milestone 4 Representation 8% Milestone 3 5% 39% Milestone 2 11% 5% Benchmark 1 13% 11% Zero 0 4 -Year Institutions Milestone 4 Milestone 2 Zero 0 33% 23% 14% 24% 13% Communication 5% 20% 31% 7% 18% 29% Assumptions 1% 43% 32% 16% Application/Analysis 22% 10% 13% 53% 1% 32% 37% 16% 26% Representation 4% 8% Milestone 3 Benchmark 1 21% 39% Interpretation 16% 13% 34% 39% 8% 0% 5% 29% Communication Assumptions Application/Analysis 20% 16% 22% 20% 56% 23%

0, 5 Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Critical Thinking at 2 -Yr Colleges 0, 25 0, 15 0 0 Explanation of Issues Evidence -0, 13 -0, 25 -0, 5 Context and Assumptions 0 Student's Position -0, 11 Conclusion

Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Critical Thinking at 4 -Yr Colleges 0, 5 0, 47 0, 45 0, 39 0, 42 0, 25 0 Explanation of Issues -0, 25 -0, 5 Evidence Context and Assumptions Student's Position Conclusion

Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Written Communication at 2 -Yr Colleges 1 0, 75 1, 05 1 0, 770000001 0, 53 0, 5 0, 25 0, 12 0 Context and Purpose -0, 25 -0, 5 Content Development Genre and Conventions Sources and Evidence Syntax and Mechanics

Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Written Communication at 4 -Yr Colleges 1, 37 1, 25 1 0, 75 0, 25 0, 19 0 -0, 13 -0, 25 -0, 5 Context and Purpose -0, 16 -0, 11 Content Development Genre and Conventions Sources and Evidence Syntax and Mechanics

Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Quantitative Literacy at 2 -Yr Colleges 0, 5 0, 28 0, 25 0, 11 0, 0700000001 0 Interpretation Representation Application and Analysis -0, 21 -0, 25 -0, 5 Assumptions -0, 13 Communication

Differences in Average Outcome Scores from 2016 to 2017 in Massachusetts Quantitative Literacy at 4 -Yr Colleges 0, 5 0, 25 0 Interpretation Representation Application and Analysis -0, 25 Assumptions Communication -0, 35 -0, 44 -0, 52 -0, 5 -0, 75 -1 -0, 850000001 -0, 59

How Mass. 2017 Compares to Project Level - Critical Thinking (2 -year colleges) 0, 5 0, 25 0, 17 0, 11 0, 14 0, 12 0, 0700000001 0 Explanation of Issues -0, 25 -0, 5 Evidence Context and Assumptions Student's Position Conclusion

How Mass. 2017 Compares to Project Level - Critical Thinking (4 -year colleges) 0, 5 0, 25 0, 17 0, 21 0, 19 Context and Assumptions Student's Position Conclusion 0 Explanation of Issues -0, 25 -0, 5 Evidence

How Mass. 2017 Compares to Project Level - Written Communication (2 -year colleges) 0, 5 0, 25 0, 12 Context and Purpose -0, 5 0, 17 0, 06 0 -0, 25 0, 17 0, 0099999998 Content Development Genre and Conventions Sources and Evidence Syntax and Mechanics

How Mass. 2017 Compares to Project Level - Written Communication (4 -year colleges) 0, 5 0, 25 0 0 Context and Purpose -0, 25 -0, 5 0, 0299999998 0, 0500000003 -0, 11 Content Development Genre and Conventions Sources and Evidence Syntax and Mechanics

How Mass. 2017 Compares to Project Level - Quantitative Literacy (2 -year colleges) 0, 5 0, 36 0, 25 0, 23 0, 11 0 -0, 25 -0, 05 Interpretation Representation -0, 24 Application and Analysis Assumptions Communication

How Mass. 2017 Compares to Project Level - Quantitative Literacy (4 -year colleges) 0, 5 0, 25 0, 11 0, 01 0 Interpretation -0, 25 -0, 03 Representation Application and Analysis -0, 069999999 Assumptions -0, 05 Communication

Suggested Action Items • Writing: • More focus on Syntax & Mechanics and Sources of Evidence (Written Communications) in the 1 st and 2 nd years • More activities related to Written Communications in 3 rd and 4 th year courses, programs, and majors (except in Syntax and Mechanics) • Resume or enhance existing “writing across the curriculum” activities • Faculty training on sources and evidence • Quantitative Literacy: • Increase activities related to Quantitative Literacy in 3 rd and 4 th year courses, programs, and majors • Employ quantitative literacy across the curriculum in addition to general education courses • Provide training for faculty on how to employ Quantitative Literacy in curriculum and assignment design at all levels • DHE to launch regional Quantitative Literacy faculty workshops in 2018

Suggested Action Items • Artifact & Assignments: • Focus on a high degree of fit between the VALUE Rubric and the assignment design, especially in writing and quantitative literacy • Continue to be selective in artifacts submitted and alignment with VALUE Rubrics

How Do We Address the Issues of Quality and Equity Without the MSC Going Forward? Dr. Lane Glenn President, Northern Essex Community College Chair, Task Force on Statewide Assessment January 19, 2018

How do we address the issues of quality and equity without the MSC going forward? • Forego the MSC/VALUE Institute due to cost • Each institution addresses the issue on an ad hoc basis • Continue to participate in the MSC/VALUE Institute on a periodic basis (e. g. , every three years) on a shared cost basis between the DHE and the institutions • Other approaches?

How do we assess the issues of quality and equity without the MSC going forward? • Forego the MSC/VALUE Institute due to cost: • No method to assess institutional quality and address equity • Weakens the ability of institutions to demonstrate the assessment of educational quality • Loss of statewide perspective and the ability to effect policy • Inability to fully meet state legal (Chapter 15 A, Section 32) for assessing quality • Loss of national perspective

How do we address the issues of quality and equity without the MSC going forward? • Each institution addresses the issue on an ad hoc basis: • Could still meet institutional assessment needs • Saves the cost of the VALUE Institute (projected to be at least $4, 500 per institution for 2018 -2019) assuming the use of lower cost vendor or a local solution • Forego statewide perspective and the ability to impact policy • Weakens the state’s ability to meet its legal and performance measurement requirements • May forego national perspective if a local approach is employed versus the VALUE Institute

How do we assess the issues of quality and equity without the MSC going forward? • Continue to participate in the MSC/VALUE Institute on a periodic basis (e. g. , every three years) on a shared basis between the DHE and the institutions: • Enables institutions to meet their institutional assessment needs • Enables the state to meet its legal and performance measurement requirements • Provides institutions time to implement changes before assessing the impact • Enables the state to address statewide policy issues • Shares the cost and responsibility between the institutions and the state

How do we assess the issues of quality and equity without the MSC going forward? • Other Approaches? • Each institution do their own approach, be it VALUE Institute, another vendor, or a local solution: • Issues of consistency • Issues of data comparability • Each institution provide their local data to DHE to be aggregated and reported on a statewide basis: • Issues of system comparability • Issues of institutional data privacy • DHE will need to obtain a technical platform to process the data an to add technical/statistical expertise to generate reporting • Issues of additional costs