Transformative Assessment Case Study June 20 2003 Gloria

“Transformative Assessment Case Study” June 20, 2003 Gloria M. Rogers, Ph. D. Rose-Hulman Institute of Technology Copyright [Gloria M. Rogers, Rose-Hulman Institute of Technology] [2003]. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Rose-Hulman Institute of Technology S S S Terre Haute, Indiana 1800+ undergraduate students B. S. degrees in engineering, science, and mathematics 85%+ engineering students 18% female

Catalyst for assessment (Note absence of the word ‘transformative’) S S S Regional accreditation ABET accreditation Absence of coherent planning process SAlthough the need to have an “assessment plan” was the catalyst for action, assessment ‘emerged’ from the planning process —can’t assess what you don’t have

Assessing Student Learning S S S Five strategic goals linked to mission statement (Input, Quality, Climate, Learning Outcomes, Resources) Created dashboard indicators for four ‘non-outcome’ goals Defined our assessment question for student learning outcome “Can students demonstrate the performance criteria at a level appropriate for a student who will graduate from Rose-Hulman? ” S Defined our assessment process

Principles that guided our assessment/evaluation processes S AAHE 9 Principles of Good Practice for Assessing Student Learning (www. aahe. org/assessment/principl. htm) S Faculty criteria for choosing assessment methods SDirect measures of student learning SResults meet external demands SNon-intrusive (students and faculty) SEasy

Student portfolios primary assessment tool S Ros. E-Portfolio developed SStudent Module 1997 SFaculty Rating module 1998 SCurriculum Mapping module 1999

Example of Results Is the submitted material at a level expected of a student who will graduate from RHIT? YES 1. Identify the readers/audience 2. Technical Content 3. Audience response 4. Grammatically correct

Outcome Explicit. This outcome is explicitly stated as being a learning outcome for this course. Demonstrate Competence. Students are asked to demonstrate their competence on this outcome through homework, projects, tests, etc. Formal Feedback. Students are given formal feedback on their performance on this outcome. Not covered. This outcome is not addressed in these ways in this course. Outcome 1. Recognition of ethical and professional responsibilities. view criteria or make a comment (optional) Outcome Explicit Demonstrate Competence Formal Feedback Yes *Yes 2. An understanding of how contemporary issues shape and are shaped by Yes of code Yes 1. mathematics, Demonstrate knowledge of science, and engineering. ethics view criteria or make a comment (optional) the ethical dimensions of a 3. An ability to recognize the 2. role Evaluate of in their professionals in the global society problem and to Yes discipline Yes understand diverse cultural and humanistic traditions. view criteria or make a comment (optional) 4. An ability to work effectively in teams. view criteria or make a comment (optional) Yes Not Covered

Transformation? Emerged from institutional vision, mission, culture q and context Focused on learning outcomes, processes and purposes Qualitative in nature Based on iterative, collaborative framework…explains relationship between teaching/learning… Large-scale, systemic and contagious Enabled by intelligent and appropriate technology Informed by assessment and a commitment to data-driven decision making. q q q q

Assessment Purpose Application of Findings Assessing Transformation Data Acquisition and Analysis Dissemination

Yr. 2: 1)Assessment process emerged from the planning Yr. 5: Departments use curriculum map to drive discussions process and involved key campus constituents and external Yr. 7: Faculty take ownership of process; significant increase and decision-making about alignment of courses tomore outcomes. Yr. 6: Yr. 4: VP Curriculum actively supporting map implemented; processes. new Department conversations Heads about Yr. 3 Rating rubrics refined, performance criteria stakeholders. No impact on decision making. 2) Data in studentoutcomes; participation; faculty-led seminars well attended; Faculty-led seminars instituted inform new faculty andon ‘officially’ student sign on tomap data using used Ros. E-Portfolio totomake course data improvements as the focused, curriculum developed. Focus was primarily documenting student learning from multiple sources and Departments including data in theirfor department assessment engage faculty in discussions related to student primary —someother progress source for but data still collection isolated. Faculty ‘soft leaders six’. Timeline emerge. process and engaging more faculty. evaluated by multi-disciplinary faculty 3) Results not used planning; institutional budget line for portfolio process and outcomes and Ros. E-Portfolio process. developed that focused the data collection effort. to reshape teaching and learning beyond isolated areas; 4) rating. Results only used to change the assessment process itself.

Assessmen t Purpose Applic ation o f Findin gs Assessing Transformation Data Acquisition and Analysis Diss emin ation

Barriers to change S S S Case for action was externally driven Unfamiliar processes (distinctions between classroom and institutional assessment) Uninformed faculty Uninterested students Uninvolved administrators

Assessing Change S S S S Faculty engagement (how many, in what ways) Student participation (who, in what ways) “Whine” meter Budgetary support Data requests Department agendas (changing conversations) Curriculum changes based on assessment results

Assessing Technology S Pilot test each module before implementation SStudents SFaculty SData users S Embedded assessment in rating module (“log”) SFocus groups with raters following each rating session S S Embed assessment/feedback in student module Embed feedback in curriculum map

Lessons Learned S S S S S Define your assessment question! Develop robust assessment processes appropriate to the question (including closing the loop) You can’t do everything Pick your battles More data are not better Be clear about carrots and sticks Have to have a ‘technology owner’ (build confidence) Faculty leadership/administrative support LEADERSHIP, LEADERSHIP (You’re da Dean)