FYS Assessment Where We Are What Weve Learned

FYS Assessment Where We Are & What We’ve Learned So Far

Fall 2015 FYS Instructor Survey O 76% include participation as part of grade (median % = 10%) O 90% using Wyo. Courses (59% using extensively, 31% occasionally) O # of assignments – median 10, average 11. 5 O Wide range of assignments in FYS courses O Research papers, reflective papers, other papers, group projects, group presentations, annotated bibliographies, library research/citation exercises/library modules, exams and quizzes, case studies, letter writing, debates, service learning/community engagement projects, interviews, speech, poster, video, blog, lab exercise, group-led discussion, resume, magazine cover, budget exercise, threaded discussion analysis, personal leadership philosophy, autobiography, etc.

Distribution of SLOs Preliminary analysis = We’ve got them covered. SLO 1 SLO 2 SLO 3 SLO 4 SLO 5 SLO 6 Major Project 17% 16% 17% Assign 2 18% 16% 17% 15% 17% Assign 3 18% 18% 13% 14% 18% Assign 4 15% 18% 17% 17% Assign 5 17% 16% 17% 17% Assign 6 20% 18% 16% 15% 17%

Fall 2015 Data Collection O Draft FYS rubric and SLOs made available to faculty on Wyo. Courses O 18 instructor volunteers submitted example of student work O Instructors asked which SLOs were covered by assignment O Most, but not all assignments were submitted via Wyo. Courses O About 450 pieces of student work collected

Spring 2015 Examination of Student Work O Recruited group of 8 FYS instructors to examine student work O Used Wyo. Courses to evaluate student work (each person had 2 -4 assignments) O Group discussion of process itself O Revised rubric O Read assignments and applied rubric O Met this month to discuss findings and next steps

Old vs. New Rubric O Changed header names and assigned numerical values O Revised language (ex. 2 vs. 3) O Some inconsistent (comparing apples to oranges) O Removed aspects of outcomes that could not be reasonably evaluated by out readers after the fact

Quick Results O SLO 2 A: O SLO 2 B: O SLO 3: O SLO 4: O SLO 5: O SLO 6 A: O SLO 6 B: Avg 1. 53, Med 2 Avg 1. 11, Med 1 Avg 1. 36, Med 1 Avg 1. 68, Med 2 Avg 1. 10, Med 1 Avg 1. 88, Med 2 Avg 1. 64, Med 2 O 479 duplicated pieces of student work rated. O SLO 6 A and 6 B rated the most. O SLO 5 rated the fewest.

Themes We Discussed O Still have about 250 pieces of student work to rate O Rubric seems to be ok to proceed O Difficult for raters to see all of the SLOs in various O O O assignments (e. g. instructors said they were in assignment, but readers couldn’t see it clearly) Difficult to evaluate 4 -6 SLOs per assignment Sometimes assignment details weren’t always clear Even though raters struggled a bit with ratings, overall ratings were fairly consistent between raters (O vs. 1 or 1 vs. 2 or 2 vs. 3). But will examine further. Would be better that raters talk and ask questions of each other instead of working independently Expertise is needed to develop consistency in ratings over time

Recommendations O Continue to use same rubric O Examine inter-related reliability O Develop template for reporting results O Statistics O Qualitative themes the readers see O Areas of focus for improvement for future FYS courses O Professional development opportunities O Regular communication with FYS instructors O University-wide recommendations to “close the loop”

Summer Work O 4 volunteers from initial group to evaluate remaining FYS student work from 2015 -16 O Continue to discuss identified issues O Decide on a report template, analyze results, and draft first report O Working toward completion of HLC Quality Initiative Report due in Jan 2017

Fall 2016 Collection of Student Work O FYS instructors asked to submit example of student work for class O One example focusing on one or two SLOs O Examples of student work must be submitted on Wyo. Courses because we have a new LTI system built especially for assessment purposes at UW

Directions O Assignment must be in assignment box O Student work uploaded by students O Instructor tags assignment for assessment O Instructor attaches rubric with SLO(s) – one or two O Rubric is ready made. Instructor attaches and deletes all but selected SLOS O Specific instructions will be coming out this summer to FYS instructors (sometime in June) O In order to receive USP designation, you are required to participate in assessment process