Assessing Learning Outcomes Thinking Critically about Critical Thinking
Assessing Learning Outcomes: Thinking Critically about Critical Thinking and Written Communication Skills Vera Beletzan, Paula Gouveia, and Kim Robinson Humber College
Agenda 1. Overview of Research Study 2. Experience the Critical Thinking Assessment Tool 3. Discussion and Questions 2
Humber: A Canadian Polytechnic 3
4
Phase 1 (2013 -14): Humber & HEQCO-LOAC Objectives: • Develop LO assessment tool • Pilot tool to measure and assess students’ critical thinking and written communication (CTWC) skills in their written course work • Assess tool for usability • Revise tool in light of faculty feedback 5
Phase 2 (2014 -16): Cross-sectional and Longitudinal Analyses 1) What is the validity of the assessment tool? 2) What is its reliability when used repeatedly? 3) Where do students experience greater levels of skill achievement in CTWC? 4) Are students making gains in CTWC learning outcomes over time? 6 5) What is the usability and scalability of the revised tool?
Cross-Sectional Analysis Three CTWC learning experiences targeted for comparison of skills achievement: - EXPLICIT = cross-college reading & writing courses (WRIT) - IMPLICIT = vocational - Police Foundations Program (PFP) - CONTENT-BASED = General Arts and Sciences (GAS) critical thinking course (GCRT) 7
Participants: Cross Sectional-Analysis 650 students from 3 of Humber’s schools: Business (BUS) (19%) Liberal Arts and Sciences (LAS) (21%) Social and Community Services (SCS) (57%) Participating Students by Course Frequency 300 200 100 257 123 137 54 0 WRIT 100 WRIT 200 GCRT 100 PFP 211 8
1) What is the validity of the assessment tool? • Content and Face Validity: Substantial • Concurrent & Convergent Validity: CTWC scores correlated with a variety of assessments (TER for concurrent / final exam and course grades for convergent) = Significant, positive and large / moderate correlation 9
2) What is the reliability of the tool? • Inter-rater Reliability (IRR): Consistency of faculty ratings of total CTWC skills scores = excellent However, IRR for individual CTWC categories ranged from excellent/good to fair/poor • Internal Consistency: Excellent to good 10
3) Where do students experience greater levels of skill achievement in CTWC? Total CTWC Skills Scores by Course 25 Median Score 20 15 10 5 0 11 WRIT 100 WRIT 200 GCRT 100 PFP 211
Longitudinal Analysis: 4) Are students making gains in learning the CTWC skills over time? • CTWC skills assessed in mandatory, sequenced cross-college WRIT courses over two semesters to determine students’ gains over time • Participants: 32 students from BUS (34%) and SCS (63%) 12
CTWC Skills Scores by Time Total CTWC Skills Integration of Writer's Ideas Comprehension 0 10 20 Mean Score Time 2 - Winter 2015 WRIT 200 Time 1 - Fall 2014 WRIT 100 13 30
5) What is the usability and scalability of the tool? • 46 faculty members piloted the tool between 2014 and 2016: LAS (76%), BUS (14%) and SCS (10%) • 74% of faculty who used the tool agreed it was easy to use • 58% of faculty agreed it provides meaningful information to students on their CTWC skill 14 level and how to improve it over time
Recommendations I • Teach CT explicitly and as a skill • Teach CT consistently and build over time • Position courses where CT taught explicitly (e. g. WRIT) strategically throughout a program for maximum exposure and ample time to practice 15
Recommendations II • Cross-institution scale-up is possible, but the tool is most effective in courses where CTWC skills are taught explicitly (e. g. WRIT) • Significant faculty training is needed to establish a common understanding of essential skills, their importance and their assessment • Training will improve overall reliability and hence accuracy and validity 16
Recommendations III • Finally, to ensure the tool is meaningful, students must understand which skills are being assessed and why • For significant gains, students’ skills must be evaluated throughout their programs, with ample opportunity for practice 17
Experience the Assessment Tool STEPS: 1. Read the assignment instructions. 2. Read the student response (paper). 3. Assess the response using the CTWC Assessment Scorecard. 4. Discuss your experience in groups. 18
Any Final Questions / Comments? 19
Thank you. vera. beletzan@humber. ca paula. gouveia@humber. ca kim. robinson@humber. ca
- Slides: 20