Constructive guidance for learning Standardsbased assessment and exemplars

Constructive guidance for learning: Standards-based assessment and exemplars Dr Graham Hendry University of Western Sydney Thursday 6 August 2009

Constructive alignment is about achieving consistency in the learning environment, for active knowledge creation Biggs, J. & Tang, C. (2007). Teaching for quality learning at university (3 rd ed. ). Maidenhead, UK: Open University Press/Mc. Graw-Hill

Constructive alignment Learning outcomes state what you want students to be able to do Teaching and learning activities Assessment Teaching and learning Assessment tasks activities actively validly measure engage students in whether students’ can learning how to do do these things Biggs, J. & Tang, C. (2007). Teaching for quality learning at university (3 rd ed. ). Maidenhead, UK: Open University Press/Mc. Graw-Hill

Bloom’s revised taxonomy

The timing and amount of assessment is important to help students manage their time and effort for knowledge creation … Excessive workload = Surface approach

To help students manage their time and effort … þ Coordinate the timing of assessment tasks with assessment in other units to avoid periods of overload þ Use assessment tasks which are authentic for the discipline or profession and which focus on students’ future needs = Relevance þ Stage tasks > the next task builds on the previous one, eg, In Law, a letter of advice to a client in plain English builds on a scholarly group research task about the relevant law/client’s rights

Manage students’ expectations from the beginning …

At a minimum … • Describe assessment tasks for the unit – How tasks relate to the most important learning outcomes – How tasks relate to students’ future discipline/professional needs • Describe how students will be marked/graded (criteria and standards) e. g. , marking sheet – What are things teachers are looking for that students will be marked on? … the criteria – What is the quality of these things they are looking for? What do students have to do to get a pass … a distinction? … the standard(s)

None of this matters unless we can engage students with written descriptions

Project in School of Law Second semester 2008 First semester 2009 Based on research at Oxford Brookes University, Business School, UK Rust, C. , Price, M. & O’Donovan, B. (2003). Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education, 28, 2, 147 -164.

• Step 1 – Exemplars (past Fail, Pass and Distinction letters of advice to a client) were handed out in class, and students were asked to mark them in groups using the marking sheet in the unit Learning Guide • Step 2 – Groups reported back on the reasons for their grading decisions • Step 3 – Lecturer revealed the correct grades and explained why these grades were given based on criteria and standards, affirming and/or challenging students’ understanding

Results, where a score of 5 is strong agreement The process of marking the three example letters in class was useful for completing my letter assignment (Campus A only) 4. 49 It was easy to tell which example letters were good and poor 4. 33 I used the example of a good letter to structure my letter assignment 4. 22 Seeing the examples of letters gave me the confidence to do a good job on my letter assignment 4. 21 The examples of letters showed me how to do the letter assignment 4. 19 The marking sheet made it clear what was expected for a well-written letter assignment 4. 02 The marking sheet was easy to understand 3. 96 The marking sheet made me feel confident that my markers were marking fairly 3. 71 The marking sheet showed me how to do the letter assignment 3. 51 Please cite as: Hendry, G. D. , Bromberger, N. , & Armstrong, S. (in press). Constructive guidance and feedback for learning: The usefulness of exemplars, marking sheets and different types of feedback in a first year Law subject. Assessment & Evaluation in Higher Education.

Results, open comments Please tell us what was most helpful for guiding you in doing your letter assignment in this unit. Why was it helpful? • “The examples of the previous years students’ letters were most helpful as it gave us examples of what a good and what a bad letter contained” • “The example letters as this helped to give a base of what was expected” • “The example letters. It was helpful because it demonstrated to us how to structure our letter (format) and also what’s considered as good and what’s bad” • “The three example letters and the discussion of them in class helped me understand how one might look and what the marker would be looking for” • “Being given example letters that were of varying qualities and having to analyse each one to determine the best/the elements required for our individual letter” • “The most helpful part was looking over the example letters to gain an idea of how to structure and word the letter well”

“I was like ‘oh my God this is…’, I would not have thought to do it that way, there is no way I would have done [it] that way, I would have definitely done it the first way, checking all the legislation and showing what I know. Just like [secondary] school you know you just want to show much you know. ” (First-year student, Law)

“exemplars convey messages that nothing else can” Royce Sadler, Professor of Higher Education Griffith University

Between campus/teacher comparisons, 2008 Campus A teacher asked students to mark exemplars; facilitated a class discussion N=39 Campus B teacher did not discuss exemplars; made them available online only N=16 Guidance provided by the exemplars Students at Campus B rated guidance provided less highly (z = -2. 22, p = 0. 03) Ability to differentiate exemplars Students at Campus B found it less easy to tell which examples were good and poor (z = -2. 25, p = 0. 02) Use of the good letter Students at Campus B used the example of a good letter to structure their assignment less (z = -2. 55, p = 0. 01)

The process of marking the three example letters in class was useful for completing my letter assignment 2008 N=55 2009 N=181 4. 49 3. 72 The examples of letters showed me the structure expected in a good letter assignment 3. 60 The process of discussing the three example letters in class with the teacher was useful for completing my letter assignment 3. 57 The examples of letters showed me the style of language expected in a good letter assignment 3. 56 I used the example of a good letter to structure my letter assignment 4. 22 3. 47 Seeing the examples of letters gave me the confidence to do a good job on my letter assignment 4. 21 3. 38 The examples of letters showed me how to do the letter assignment 4. 19 3. 38 The marking sheet made it clear what was expected for a well-written letter assignment 4. 02 3. 26 The marking sheet was easy to understand 3. 96 3. 26 The marking sheet showed me how to do the letter assignment 3. 51 3. 15

Between teacher comparisons 2009, where higher rank = stronger agreement The marking sheet made it clear what was expected for a well-written letter assignment Teacher A mean rank N=50 (discussed exemplars with class) Teacher B mean rank N=38 (gave out exemplars and focussed on what not to do) Teacher C mean rank N=51 (did not discuss exemplars) 50. 19 37. 01 p < 0. 01 43. 24 p < 0. 04

Teacher A (discussed exemplars with class) Please tell us what was most helpful for guiding you in doing your letter assignment in this unit. Why was it helpful? • “Marking the three examples was most helpful, as we got to see what we thought were good answers compared to X’s. ” • “Discussing the letter structure and style in the workshop was most helpful. Gave you a chance to ask questions. ” • “The most helpful aspect in doing my letter assignment was the workshop where we looked at example letters. It was very useful because we analysed the letter in great depth pointing out the weaknesses and strengths of the letter. The workshop provided a good guide in regards to writing a letter for legal purposes. ” • “Discussing and marking the three examples was most helpful, as it put us in the markers position, therefore shedding light upon what we needed to do to excel in this task. ”

Teacher B (gave out exemplars and focussed on what not to do) Please tell us what was most helpful for guiding you in doing your letter assignment in this unit. Why was it helpful? • “Nothing. I felt we were only shown what NOT to do in my class, not what actually should be done. ” • “I think discussing the letter structure was most helpful. Nevertheless it would have been most beneficial if we were given more guidance in this area. ” • “Until discussing the letter, I wasn’t aware that we needed a letter head, so that was good. X showed us letters ranging from a 1 – 20 mark to a 10 – 20. We didn’t see any HD students. ” • “Showing me not what to do. ” • “The example letters because I’d never seen one before so it gave me a structure to work off. ”

Teacher C (did not discuss exemplars) Please tell us what was most helpful for guiding you in doing your letter assignment in this unit. Why was it helpful? • “I don’t think there is anything useful about the example of the letter. I just used it to follow the format of the letter. ” • “Which letters? I have to find for myself. Examples that I wasn’t sure if it was correct!!. ” • “Statement 3 – Didn’t discuss. ” • “Nothing. This assignment, just like the case note, needed more time dedicated to how to do it and what was expected. ” • “The example of good letter was very helpful to me as it gave me the guidelines of what is expected of me. ”

Feedback that focuses on how students can go about improving their skills is more usable and valued by students … Walker, M. (2009). An investigation into written comments on assignments: Do students find them usable? Assessment & Evaluation in Higher Education, 34(1), 67 -78.

Is the use of exemplars spoon feeding or teaching students how to fish? “Give a person a fish and you feed them for a day. Teach them how to fish and you feed them for a lifetime. ” Lao Tzu, Philosopher

Effects of ‘marking class’ on students’ performance Participants’ mean grade Nonparticipants p Effect size Class 1 baseline 58. 6 57. 8 0. 56 0. 08 Class 1 after workshop 59. 8 54. 1 <0. 001 0. 6 Class 1 one year later 57. 9 51. 3 <0. 005 0. 55 Class 2 baseline 58. 7 55. 7 0. 06 0. 31 Class 2 after workshop 59. 9 52. 9 <0. 001 0. 69 Source: Rust, C. , Price, M. & O’Donovan, B. (2003). Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education, 28, 2, 147 -164.

Types of criteria and standards ‘Objective’ criteria ‘Subjective’ criteria 1. Regulative = regulate the assignment (length, referencing) 3. Prescriptive = prescribe qualities of the work (coherence, readability) 2. Logical = refer to valid reasoning (correct/incorrect) 4. Constitutive = constitute features of the discipline (historical meanings, scientific methodology) Standards are well-defined Standards are ‘matters of degree’ Source: Sadler, D. R. (1983). Evaluation and the improvement of academic learning. Journal of Higher Education, 54(1), 60– 79