Designing Grading Rubrics to Assess Student Learning The

Designing Grading Rubrics to Assess Student Learning The DIY (Do It Yourself) Workshop Mary T. Blackinton PT, MS, Ed. D Director Transition DPT Program Chair, PT Outcomes Committee Nova Southeastern University Dr. Mary Blackinton, 01 -19 -08

Agenda n Introduction to Grading n n n Analyzing Grading Rubrics n n Small group discussion Developing Rubric n n n Myths & Principles of Effective Grading Choosing an Assignment for a Rubric Definitions Primary Trait Analysis Creating & Using Rubrics n n Design Pitfalls Dr. Mary Blackinton, 01 -19 -08

Objectives n Upon completion of the workshop, faculty will be able to: n Describe the role of grading rubrics in: n n n n Clarifying teacher expectations Evaluating student learning Providing detailed feedback Improving instruction Performing outcome assessment Evaluate the strengths and weaknesses of grading rubrics as a student assessment tool in health professions education Integrate the principles of primary trait analysis (PTA) into designing a rubric for an assignment or performance skill Create a grading rubric for. Blackinton, an assignment, project, or exam Dr. Mary 01 -19 -08

Grading Challenge # 214: Student’s Perceptions Dr. Mary Blackinton, 01 -19 -08

Why do we Grade? n Grading 1 n The process used by faculty to assess student learning via assignments and exams, including: þ Relating test items or assignments to learning objectives þ Establishing criteria/standards (SAFETY) þ Helping students acquire needed knowledge/skills þ Facilitating student motivation þ Giving feedback about performance þ Communicating about what has been learned þ Using results to influence teaching and curriculum BE Walvoord, VJ Anderson (1998). Effective grading: A tool for learning and assessment. Jossey-Bass: San Francisco 1 Dr. Mary Blackinton, 01 -19 -08

Student Assessment in Health Professions n Faculty charged to assess competency in patient care n n n Professionals integrate 3 domains of learning n n Responsibility to protect public Other stakeholders: accrediting bodies, profession, students, parents… Cognitive-thinking/reasoning Affective-feeling, valuing, ethics Psychomotor-doing, manipulating, performing, administering Assessment strategies reflect demands professional ed n n Must include more than multiple choice assessment to determine learning Often have multiple faculty grading student performance Dr. Mary Blackinton, 01 -19 -08

Myths About Grading n Grading: n n n n Is a necessary EVIL Must be objective Should be easy Should not take detract from teaching Use knowledge of experts to evaluate novices Is assessment only, doesn’t impact learning With clear expectations means you are “spoon-feeding” Cannot be used in program assessment Dr. Mary Blackinton, 01 -19 -08

Rubrics as Assessment of Program Outcomes Learning Outcome Direct Measure Assessment Method Students will be able to select valid/ reliable tests/measures to assess patient outcomes Assignment 2, Neuromuscular I: Given case, students select tests/measures & provide rationale based on patient needs & psychometric properties Rubric identifies ability to ascertain validity & reliability of tests and measures by reading relevant literature, and applying to specific needs of a patient (see sample) Dr. Mary Blackinton, 01 -19 -08

Principles of Effective Grading 1 Appreciate the complexity of grading Substitute judgment for objectivity Distribute time effectively Be open to change Listen & observe Communicate & collaborate with students Integrate grading with other key processes Seize the teachable moment (feedback) Make student learning the primary goal Be a teacher FIRST, gatekeeper LAST Encourage learning-centered motivation Emphasize student involvement 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 1 Walvoord &Anderson (1998) Dr. Mary Blackinton, 01 -19 -08

Types of Grading in Health Professions Student Assessment Methods n n n Student. Constructed Responses n n n n 2 LC Multiple choice examination Short answer questions on tests Essay questions Annotated bibliographies Literature reviews Case Studies Oral Examinations Practical/Performance Examinations Clinical Assessments Journals Portfolios Lab Conclusions Jacobs, CI Chase (1992). Developing and using tests effectively. Jossey-Bass: San Francisco Dr. Mary Blackinton, 01 -19 -08

Methods of Grading Student-Constructed Responses Method Advantage Norm Referenced: Little work up front, categorize work into recognizes faculty as A, B, C, D ‘expert’, flexible Checklist: list of Assignment directions criteria to include match checklist, not (introduction, difficult to prepare research question…) Disadvantage -Potential for bias -Less opportunity for learning, vague -Usually lacks descriptions -Lists + traits or behaviors, no negative Grading Rubric: Explicit expectations, -Takes time to criterion referenced, better feedback, greater construct, may need describe performance inter-rater reliability, to modify after 1 st run expectations & links to performance Dr. Mary Blackinton, 01 -19 -08

What are Assessment Rubrics? Small Group Discussion In small group, look at the sample rubric packet. Then, answer the following: n Discussion Questions Which rubrics seem the MOST clear to you? Why? n Which rubrics are not clear? Why? n What was common among the rubrics you viewed as clear? n Based on samples, how do you define assessment rubrics? n Dr. Mary Blackinton, 01 -19 -08

What is a Grading Rubric? n n n Method of “articulating expectations for an assignment by listing the criteria, or what counts, & describing levels of quality from excellent to poor” 3 Type of assessment that specifies gradations of quality from excellent to poor 4 A criterion-referenced method of grading using highly specific grading criteria that are linked to objectives HG Andrade, Y Du (2005). Students perspectives on rubric-referenced assessment. Practical Assessment, Research & Evaluation, Vol 10 (3). 3 HG Andrade ((2005). Teaching with rubrics: The good, the bad, and the ugly. Coll Teaching, 53 (1) Dr. Mary Blackinton, 01 -19 -08 4

What is a Grading Rubric? Gradation: excellentpoor Categories important to the teacher/class Performance Level 3 Level 2 Level 1 Dimension 1 3 pts: describe 2 pts 1 pt Dimension 2 Dimension 3 4 pts 2. 5 pts 1 pt 3 pts 2 pts 1 pt Dimension 4 2 pts 1 pts 0 pts Weighted Points Dimensions also called criteria Dr. Mary Blackinton, 01 -19 -08

Criteria 1 Criteria 2 Criteria 3 Level 1 3 pts: describe 2 pts 1 pt Level 2 4 pts 2. 5 pts 1 pt Level 3 3 pts 2 pts 1 pt Level 4 2 pts 1 pts 0 pts Dr. Mary Blackinton, 01 -19 -08

How do Students Perceive Rubrics? n Andrad & Yu, 2005 n Investigated how students use grading rubrics n n Students reported that they used rubrics to n n n Focus group / qualitative design Help them determine faculty expectations To plan an approach to assignment Check/revise work before handing in Help reflect on their learning-see strengths/weaknesses clearly Perceived results of rubric use n n n Better, fairer grades Improvements in quality across classes Less anxiety Dr. Mary Blackinton, 01 -19 -08

Using Primary Trait Analysis 1, 4 To Develop a Grading Rubric Rationale n Analyze traits / characteristics of student learning and then clearly articulate them, leading to: n Assignment specific n Explicit criteria n Adds objectivity & consistency to holistic scoring n Lets students know in ADVANCE how will be graded 1 Walvoord &Anderson (1998) 4 R Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell (Eds), Evaluating Writing: Describing, measuring, judging. Urbana IL: National Council of Teachers Dr. Mary Blackinton, 01 -19 -08

Constructing a Primary Trait Analysis Scale 1 1. 2. Choose assignment/test that matches course objective Identify all traits that will count for scoring in assessment 1. 3. Build scale for scoring performance, gradations of skill 1. 2. 3. 4. 5. 4. Body Language vs Thoroughness vs Accuracy Scale usually ranges from 2 -5 points Include what should be demonstrated and what should be avoided Build a range that discriminates A from A- from B+ Run scale by colleague, graduate, teaching assistant Weight items: content > spelling; accuracy > efficiency, etc Evaluate performance against criteria 1. Try scale with sample & revise as needed Dr. Mary Blackinton, 01 -19 -08

Example: Designing a Rubric n Select assignment appropriate for PTA n n Taking a Patient History (clinical performance) Identify ALL traits you look for in patient history Body language & eye contact (X-1) n Thoroughness (X-2) n Data gathering (X 1. 5) n n Weight them each trait Dr. Mary Blackinton, 01 -19 -08

Building a Scale: Body Language n Level 3 -Excellent n n n Level 2 -Good n n n At least 4 of the above behaviors noted OR, all 5 qualities demonstrated but not consistently Gives students specific behaviors to demonstrate! Identifies behaviors to avoid! Level 1 -Fair n n Consistently had good eye contact with patient Communicated at patient’s eye-level Maintained an open posture, leaning toward patient Facial expression interested, non-judgmental Confident and relaxed, not arrogant Inconsistent in several (2 -3) areas, for example, confident through some portions of history but not others; or, eye contact missing at end Level 0 -Poor n Arrogance, avoidance, lack of confidence detracted from relationship Dr. Mary Blackinton, 01 -19 -08

Building a Scale: Thoroughness n 4: Extremely Thorough n n 3: Thorough n n All items (1 -9) included but may have missed some sub-categories (ie missed 2 body systems); or, only missed 1 -2 of the 9 core areas above 2: Somewhat Thorough n n History includes all the following core areas: 1. all body systems; 2. history current illness/problem; 3. family history; 4. medications; 5. educational background / learning style; 6. lifestyle (diet, exercise, habits); 7. living/ work/social environment; 8. assistive/adaptive devices (glasses, cane, etc); 9. prior level of function Missing 3 -4 of the 9 core areas, or, missing only 1 -2 areas but was also superficial in review of body systems 1: Incomplete n Less than half of the 9 core areas were covered; incomplete data Dr. Mary Blackinton, 01 -19 -08

Building a Scale: Data Gathering n Excellent-4 n n Good-3 n n Demonstrates at least 4 of 6 skills noted above throughout the history, or, used all 5 skills but inconsistently. Fair-2 n n 1. Uses open-ended, broad questions; 2. encourages patient to tell story chronologically; 3. allows patient to talk without interrupting; 4. encourages by using phrases such as ‘tell me more’ or ‘what else? ’; 5. summarizes what was heard; 6. Approach was organized Demonstrates 3 of the core probing skills, or, 3 -4 skills used however did so inconsistently; multiple ‘missed’ opportunities to probe; disorganized at times Poor-1 n Did not consistently demonstrate probing skills, questions were narrow and rarely open-ended, interrupted or cut patient off, rarely encouraged more information with phrases such as ‘tell me more’. Dr. Mary Blackinton, 01 -19 -08

DIY: Build Your Own PTA Assignment or Test: __________________________ Related Class Objectives: ________________________ Traits Gradations Trait 1: Trait 2: Excellent Good Fair Poor Dr. Mary Blackinton, 01 -19 -08 Trait 3:

Step 4: Evaluate Performance Using Criteria n Test the Scale n n Use it with an past paper, videotape, case report, etc. What changes are needed in the rubric? Were the criteria easy to follow? n Did the overall grade reflect performance? n n Are any changes needed in description or in the point allotment? n n Did an A performance score better than a B? Is it easy? Hard? Too cumbersome? Dr. Mary Blackinton, 01 -19 -08

Applying the Rubric Distribute ( or post) rubric to students in advance n n Have conversations about expectations Ask students to attach rubric to assignment n n Helps students pay attention! Use rubric to grade n n Match written comments to phrases in rubric Revise after use n n Make changes soon after grading for next time Answer Curriculum Questions n n Did the students learn? To what degree were objectives met? If not, was it the teaching? The assignment? Background skills? What, if any, changes should be made in the class or curriculum? Rubric? Dr. Mary Blackinton, 01 -19 -08

Advantages of Grading Rubrics 1 n n n n n Save time in grading process Makes process of grading reliable/fair Clarifies expectations for students Reinforces key concepts - help faculty relate to objectives Students are participants as expectations are known Student peer review Works well if team-teaching Share across courses or over curriculum Basis for departmental/program assessment Which of these advantages pertain to your teaching in the health professions? 1 Walvoord & Anderson, 1998 Dr. Mary Blackinton, 01 -19 -08

CAUTION: Common Rubric Pitfalls n Rubric does not correspond with class or program outcomes n n Scale does not have enough gradations or levels n n Grammar = Content Too broad, not enough content described n n n Not distinguishing the A’s from the B+’s All traits are given equal weight regardless of complexity n n Example: Entire rubric focused on writing quality not content Words like ‘breadth’ and ‘depth’ used in lieu of specifics Students still not sure, other grading faculty still not clear Too long/too complicated n Faculty + students get lost in the rubric Dr. Mary Blackinton, 01 -19 -08

Classroom Assessment Technique n Write down ONE thing you will commit to incorporate regarding the use of grading rubrics n Write down ONE unanswered question Contact: maryb@nova. edu Dr. Mary Blackinton, 01 -19 -08