How Do We Define and Design Meaningful Assessment

How Do We Define (and Design) Meaningful Assessment? NATASHA JANKOWSK I, PHD. DIRECT OR, N ATIONAL INSTIT UT E FO R LEARNING OUTCOMES A SSESSMENT RESEA RCH PROFESSOR , EDUC ATIO N POLIC Y, ORGANIZA TION AND LEADERS HIP, UIU C

NILOA’s mission is to discover and disseminate effective use of assessment data to strengthen undergraduate education and support institutions in their assessment efforts. ● SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN ● DEGREE QUALIFICATIONS PROFILE ● TUNING www. learningoutcomesassessment. org

www. learningoutcomesassessment. org

Turn to neighbor and share What makes assessment good and meaningful? What are you most proud of in the past year when it comes to student learning?

Our Time Together Address three areas of design: 1. Mapping Learning 2. Assignment Design 3. Use of Results to Enhance Student Learning

Mapping

Curriculum Mapping: The Process Focused on curriculum and program learning outcomes Two-dimensional matrix representing courses on one axis and outcomes on the other Faculty identify which courses address which learning outcomes Is it an individual process or one of consensus building? If two faculty members individually mapped the curriculum would they end up with the same map? What is standard of alignment? Is one paper in one class enough?

Learning Outcomes Learning Outcome 1 Course 2 X Course 3 Course 4 X Learning Outcome 2 X Learning Outcome 3 X X X

Learning Outcomes General Education Major Courses Activities and Possible Careers Experience That Provide Support Learning General education Courses that Outcome 1 courses that address specific support the outcomes learning outcomes Cocurricular elements that support specific outcomes Possible career paths related to the map Learning Outcome 2

Alignment How do you ensure alignment between our assignments and a given learning outcome for a learning experience? How much is “enough” to be placed in a map? Does it need to be tightly aligned or loosely aligned?

Alignment Employment Co ur se s sig nm en ts Assignments Emp Ed ral Programs ent loym e Gen As Courses General Ed Programs

Selective Attention Test

Mapping Toolkit

Assignments as Assessments Faculty are working to create a curriculum that intentionally builds in integrated learning opportunities over time for students to apply and practice as well as transfer their knowledge and skills through assignments, in and out of courses.

The NILOA Initiative An online library of high-quality peer-endorsed assignments linked to specific learning outcomes. Designed by faculty, part of the intellectual work they already do, course embedded. Building on campus efforts already underway Reflecting a conception of assessment as integral to teaching and learning vs “exoskeleton” (Ewell 2013). 17

What’s a“charrette”? "Charrette" (Fr. ) means a small cart. Because architecture students once deposited their assignments in it as the cart was rolled through the studio, architects now use the word to refer to an intense creative effort in a limited time period. 18

Aussi. . . (a) In the tradition of the “atelier, ” architecture students progress through the curriculum in the company of their mentors and peers. This approach offers an interesting model for an integrated education. (b) The tradition rests on the assumption that much of your learning will come from one another. 19

Small Group Process 25 minute segments person/assignment ØPresent assignment briefly—which outcomes, what course, which students, how does it work…. ØDiscussion with group (focus on questions on feedback form) ØSave 5 minutes for written feedback ØBreath ØStart again… ØDebrief at end

Resources: Toolkit http: //www. learningoutcomeassessment. org/assignmenttoolkit. html

Assignments as Assessment Scaffolding Learning Assignment Learning Outcomes Evaluative Criteria

Fitness of Method or Approach Part of alignment is fitness of method or approach – if asking students to explain something, multiple choice test might not be the best approach, but if asking students to identify – it might be very appropriate…

Learning Outcomes How assignments connect Scaffolding Learning Assignment Evaluative Learning Outcomes Criteria Evaluative Criteria

Program View Scaffolding Learning Assignment Learning Outcomes General Education Evaluative Criteria Major

Transparency in Assignments Transparency in Teaching and Learning: https: //www. unlv. edu/provost/teachingandlearning Purpose Skills you’ll practice by doing this assignment Content knowledge you’ll gain from doing this assignment How you can use these in your life beyond the context of this course, in and beyond college Task What to do How to do it (Are there recommended steps? What roadblocks/mistakes should you avoid? ) Criteria (Are you on the right track? How to know you’re doing what’s expected? ) Annotated examples of successful work (What’s good about these examples? Use the checklist to identify the successful parts. )

Rubrics Do we share our rubrics or criteria with students and actively engage them in the review process? Rubric Criteria Student Evaluation Faculty Feedback Rubric Content Stipulate why gave Faculty stipulate score did why gave score did Stipulate what Targeted feedback they need to do to to improve advance

Transference of Knowledge and Skills We know that students have the hardest time with transferring knowledge - realizing something they learned before can be applied in another context. We have to teach them to reflect and have them apply over time.

Timing and Placement of Assignments Do we consider student movement through curriculum in terms of the placement of signature assignments? Do we also consider transfer points and convergence between general education and the major?

Assessment Cycle

Using Assessment Results

Why the lack of use? With the majority of institutions in the US engaged in assessing student learning, why is there still a large disparity of institutions that are using assessment results ?

Causal Statements The ability to make causal claims about our impact on students and their learning Institutional structures and support + student = enhanced learning

Difficulty of Causal Statements Mobility of students Untracked changes Changes in courses add up to program level change Levels at which use occurs Longer than a year cycle Loosely coupled relationships Life

Evidence-based Storytelling Evidence of student learning is used in support of claims or arguments about improvement and accountability told through stories to persuade a specific audience.

The Why: Argumentation Toulmin (2003)

But… Toulmin (2003) Evidence Claim Warrant

Theories of Change Why do we think the changes we make will lead to better outcomes? What is assumed in the changes we select as it relates to how students understand navigate higher education?

Change Trap Don’t automatically expect improved results or try to make changes before the current changes have had a chance to take effect. A change does not an improvement make.

For instance… Coverage and content Opportunities and support Intentional, coherent, aligned pathways Within each of these is the belief about a root causes – why students were not learning or not meeting the outcome and the mechanism by which the institution can help them succeed

What does good assessment look like for us here? Why do we think that we are doing, for these students, will lead to enhanced learning, at this time?

What want to know about students? What argument do you want to make about your students’ learning? What type of evidence would be necessary to make the argument? What questions do you have about student learning? What questions do you have about programs? Why do we think that we are doing, at this institution, for these students, will get us the desired results? How do we know?

Excellence in Assessment Designations National recognition program for campus assessment leaders at two levels Evaluation based on the National Institute for Learning Outcomes Assessment (NILOA) Transparency Framework Focus on campus-wide assessment – including student affairs & external stakeholders Joint project of the VSA, NILOA, and the Association of American Colleges & Universities (AAC&U)

Why did we create the EIA Designations? Faculty & instructional staff Students & Alumni Employers, subsequent institutions Student Affairs staff Accreditors, governing boards

Recap If we don’t know what we are doing or why or what we will do with it when we have it – we can’t define or design meaningful learning outcomes assessment Once we know what we are going for – we need to understand as a program how all the pieces fit together through mapping But we also need to understand how we support that development over time in the form of assignments to prepare students to achieve and demonstrate desired learning

Questions Email: niloa@education. illinois. edu njankow 2@Illinois. edu http: //www. learningoutcomesassessment. org www. assignmentlibrary. org www. degreeprofile. org