VALUEMultiState Collaborative MSC to Advance Learning Outcomes Assessment

VALUE/Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment Pilot Year Study Findings and Summary These slides summarize results from a proof-of-concept pilot study involving 59 institutions in nine states using common rubrics to assess more than 7, 000 student work products. The sample of student work in the pilot represented the near-graduation students across the participating institutions in the nine states only; therefore, the results are not generalizable for all students in each participating state or nationwide.

MSC Pilot by the Numbers • MSC states: Connecticut, Indiana, Kentucky, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Utah • 59 public institutions uploaded artifacts • By sector: – 28 four-year, including 8 research institutions – 31 two-year These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 2

MSC Pilot by the Numbers • 7, 215 pieces of student work were submitted [number of pieces of work approximates number of student participants] – Students had to be 75% of the way to completion of institutional degree requirements – 2, 642 artifacts scored twice (36. 6%) in order to measure inter-rater reliability • 1, 166 assignments were submitted [number of assignments approximates number of faculty participants] These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 3

MSC Pilot Study Student Population Sample by Gender Relative to Graduating Students in Participating Institutions These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 4

MSC Pilot Student Population of Pell-Eligible vs. Non Pell-Eligible Students These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 5

MSC Pilot Study Four-Year Student Population Samples by Race 82% of students in MSC sample were White; 80% of students in participating institutions were White 9/9/2020 These results are not generalizable across participating states or the nation in any way. Please use appropriately. 6

MSC Pilot Study Two-Year Student Population Samples by Race 77% of MSC sample were White students; 81% of students in participating institutions were White These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 7

MSC Pilot Study Student Population Sample by Age Relative to Students at Participating Institutions 9/9/2020 These results are not generalizable across participating states or the nation in any way. Please use appropriately. 8

VALUE Rubrics Valid Assessment of Learning in Undergraduate Education • Assesses student work from curriculum and co-curriculum • Provides detailed assessment of key dimensions/criteria of learning for each learning outcome • Evaluated by faculty and other educational professionals • Visible evidence of learning applied and used in multiple settings • Shared framework and standards of judgment for quality without standardization 9/9/2020 9

Faculty and Staff Responses to Usefulness of VALUE Rubrics for Assessing Student Work Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use 9/9/2020 10

Quantitative Literacy Rubric Dimensions Interpretation Ability to explain information presented in mathematical forms (e. g. , equations, graphs, diagrams, tables, words) Representation Ability to convert relevant information into various mathematical forms (e. g. , equations, graphs, diagrams, tables, words) Calculation Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized)

MSC Pilot Study Results—Quantitative Literacy Dimension 4 -Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of quantitative literacy 50% 40% 30% 20% 10% 0% 0 1 2 3 4 Application/ Communicat Interpretatio Representati Assumptions Calculation Analysis ion n on 7. 3% 27. 6% 9. 1% 11. 2% 13. 1% 10. 7% 12. 8% 5. 4% 8. 0% 5. 1% 7. 1% 33. 3% 31. 5% 19. 4% 22. 5% 26. 4% 26. 2% 31. 3% 19. 7% 47. 8% 35. 8% 38. 2% 38. 3% 17. 3% 8. 4% 18. 3% 22. 5% 19. 1% 15. 3% Total 13. 7% 8. 2% 26. 4% 35. 1% 16. 7% Note: Each work product was scored on 6 dimensions of quantitative literacy using a common AAC&U VALUE Rubric. See Slide 15 for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 12

MSC Pilot Study Results—Quantitative Literacy Dimension 2 -Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of quantitative literacy 50% 40% 30% 20% 10% 0% 0 1 2 3 4 Application/ Communicat Interpretatio Representati Assumptions Calculation Analysis ion n on 12. 4% 35. 4% 5. 9% 14. 5% 15. 5% 7. 0% 10. 0% 17. 5% 6. 9% 14. 1% 11. 2% 7. 5% 45. 0% 24. 5% 25. 0% 22. 7% 29. 6% 34. 0% 27. 8% 18. 4% 53. 2% 40. 5% 35. 4% 45. 5% 4. 8% 4. 2% 9. 0% 8. 2% 8. 3% 6. 0% Total 15. 4% 11. 3% 30. 1% 36. 4% 6. 7% Note: Each work product was scored on 6 dimensions of quantitative literacy using a common AAC&U VALUE Rubric. See Slide 15 below for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 13

Selected Findings – Quantitative Literacy Most students in both two-year and fouryear institutions were rated “ 3” or “ 4” on students’ “calculation” skills. Less than half of work products at four-year institutions and about one-third of work products at two-year institutions were rated “ 3” or “ 4” on ability to “make judgments and draw appropriate conclusions based on quantitative analysis of data. ”

Written Communication Rubric Dimensions Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s). Content Development Genre and Disciplinary Conventions Formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields (please see glossary). Sources and Evidence Control of Syntax and Mechanics

MSC Pilot Study Results—Written Communication Dimension 4 -Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of written communication 50% 40% 30% 20% 10% 0% Content Context/ Syntax/ Genre/ Sources/ Total Development Purpose Mechanics Conventions Evidence 0. 00440528634361234 0 0. 0143884892086331 0. 012987013 0. 106451612903226 0. 0305180979418027 1 14. 5% 11. 6% 11. 8% 14. 7% 16. 1% 13. 6% 2 26. 9% 29. 5% 33. 3% 30. 7% 28. 7% 30. 2% 3 35. 2% 34. 8% 33. 3% 27. 4% 33. 0% 4 22. 9% 24. 1% 18. 7% 19. 9% 17. 1% 20. 1% Note: Each work product was scored on 5 dimensions of written communication using a common AAC&U VALUE Rubric. See Slide 18 below for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 16

MSC Pilot Study Results—Written Communication Dimension 2 Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of written communication 50% 40% 30% 20% 10% 0% 0 1 2 3 4 Content Development 2. 7% 20. 0% 30. 2% 36. 5% 10. 6% Context/ Purpose 1. 4% 12. 1% 38. 8% 30. 4% 17. 3% Syntax/ Mechanics 0. 2% 20. 9% 38. 4% 32. 4% 8. 1% Genre/ Conventions 3. 2% 24. 6% 41. 5% 20. 6% 10. 1% Sources/ Evidence 12. 8% 27. 5% 27. 1% 22. 3% 10. 3% Total 3. 7% 21. 3% 35. 4% 28. 9% 10. 6% Note: Each work product was scored on 5 dimensions of written communication using a common AAC&U VALUE Rubric. See Slide 18 below for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 17

Selected Findings – Written Communication • Students at two-year colleges have students performing really well, for example: • • Nearly 50 percent of work products collected from two-year institutions were scored either “ 3” or “ 4” on “content development” in writing. About one-third of work products collected from two-year institutions were scored “ 3” or “ 4” on demonstrating the use of “sources and evidence” in writing. • Students at four-year colleges and universities are performing at higher levels overall. • • More than 50 percent of student work at four-year institutions was scored at either a “ 3” or “ 4” overall in terms of written communication One still sees much room for improvement overall, especially in students’ capacity to use “sources and evidence” to make arguments in their writing.

Critical Thinking Rubric Dimensions Explanation of issues Evidence Selecting and using information to investigate a point of view or conclusion Influence of context and assumptions Student's position perspective, thesis/hypothesis Conclusions and related outcomes implications and consequences

MSC Pilot Study Results--Critical Thinking Dimension 4 -Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of critical thinking 50% 40% 30% 20% 10% 0% 0 1 2 3 4 Conclusions/O utcomes 6. 9% 25. 5% 38. 7% 21. 9% 6. 9% Evidence 4. 9% 27. 0% 35. 2% 26. 2% 6. 6% Explanation of Context/ issues Assumptions 2. 2% 12. 1% 16. 3% 25. 8% 37. 0% 38. 6% 30. 4% 14. 1% 9. 1% Student's Position 4. 8% 33. 2% 32. 6% 24. 6% 4. 8% Total 6. 2% 26. 0% 36. 6% 23. 3% 7. 9% Note: Each work product was scored on 5 dimensions of critical thinking using a common AAC&U VALUE Rubric. See Slide 12 for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 20

MSC Pilot Study Results--Critical Thinking Dimension 2 -Year Institutional Score Distribution Frequency Percent % of student work products scored 4 -0 by faculty scorers on each dimension of critical thinking 50% 40% 30% 20% 10% 0% 0 1 2 3 4 Conclusions/O utcomes 9. 8% 37. 9% 38. 3% 10. 2% 3. 8% Evidence 10. 1% 31. 2% 37. 7% 15. 2% 5. 8% Explanation of Context/ issues Assumptions 8. 8% 11. 6% 26. 3% 40. 1% 36. 8% 32. 7% 24. 6% 15. 0% 3. 5% 0. 7% Student's Position 9. 7% 41. 8% 27. 9% 17. 6% 3. 0% Total 10. 0% 36. 3% 34. 8% 15. 5% 3. 4% Note: Each work product was scored on 5 dimensions of critical thinking using a common AAC&U VALUE Rubric. See Slide 12 for rubric dimension criteria. VALUE rubrics are available at www. aacu. org/value. This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. 9/9/2020 DRAFT 21

Selected Findings - Critical Thinking • Consistent with other national studies, many students who had earned at least 75 percent of credits toward their degrees still are not achieving high levels of important critical thinking skills. Of the three outcomes evaluated, critical thinking was the worst. • Less than one-third of student work products collected from fouryear institutions were scored a “ 3” or “ 4” on “using evidence to investigate a point of view or reach a conclusion. ” • Nearly 40 percent of work samples assessed at four-year institutions were rated only “ 0” or “ 1” on how well students “analyzed the influence of context and assumptions” to draw conclusions. • On the other hand, nearly 45 percent of work products from fouryear institutions were rated “ 3” or “ 4” in terms of how well students “explained issues” in their analysis. • In short, students can describe and explain, but lack proficiency in critically examining the assumptions, hypotheses, evidence and implications of issues and positions, i. e. applying their knowledge.

VALUE/Multi-State Collaborative (MSC) KEY CONCLUSIONS v Central to student achievement is the quality of assignments that will allow us to reap the evidence on learning v Faculty collaboration and agreement on quality of student work across and within institutions is possible and assessment results can be useful for improvement of learning v We can and need to improve what we are doing to achieve heightened proficiency among all students 9/9/2020 23