How honest are the students when assessing each

How honest are the students when assessing each other’s work? Diogo Montalvão d. montalvao@herts. ac. uk Senior Lecturer, Dipl MSc Ph. D CEng School of Engineering & Technology How honest are the students when assessing each 2012 Annual other’s work? L&T Conference, 3 rd May 1/31

Outline 1. Why student’s peer-assessment? 2. Description of a case study at IST 3. Some of the students’ works 4. Peer-assessment results 5. Conclusions How honest are the students when assessing each other’s work? 2/31

Why student’s peer-assessment? Student benefits Learn from their peers Helps them interpreting assessment criteria Helps them perceiving their own learning Encourages critical analysis of student work Increased feedback to students source: http: //www. heacademy. ac. uk/hlst/resources/a-zdirectory/peer_assessment How honest are the students when assessing each other’s work? 3/31

Why student’s peer-assessment? Teacher benefits Summative grading purposes Reduced marking loads Engagement To get students more involved and to make them take more responsibility Assess individual effort in team work source: http: //www. heacademy. ac. uk/hlst/resources/a-zdirectory/peer_assessment How honest are the students when assessing each other’s work? 4/31

Why student’s peer-assessment? Disadvantages may be encountered when students: • Lack the ability to evaluate each other • Do not take it seriously • Fear discrimination source: http: //www. heacademy. ac. uk/hlst/resources/a-zdirectory/peer_assessment How honest are the students when assessing each other’s work? 5/31

Outline 1. Why student’s peer-assessment? 2. Description of a case study at IST 3. Some of the students’ works 4. Peer-assessment results 5. Conclusions How honest are the students when assessing each other’s work? 6/31

Description of a Case Study at IST* *Instituto Superior Técnico, Technical University of Lisbon Modules: • Technical Drawing and Geometrical Modelling I & II • 1 st year students • 2 groups/module, ~50 students/module • BEng / MEng in Mechanical, Aerospace and Marine Engineering • 2010/2011 session How honest are the students when assessing each other’s work? 7/31

Description of a Case Study at IST Technical Drawing and Geometrical Modelling I & II Assessment “Other” tests CAD* assignment 10% In-class tests 30% Group project 40% Group presentations 10% *CAD – Computer Aided Design How honest are the students when assessing each other’s work? 8/31

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 Goals: • To engage the students in practice and study • Without significant increase of the marking load 0 1 Wrong Right How do we define the boundary in a binary marking scheme for drawings? This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept. How honest are the students when assessing each other’s work? 9/31

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 Goals: • To engage the students in practice and study • Without significant increase of the marking load 0. 5 1 Wrong Acceptable Very good How to differentiate a student that does not deliver the assignment from the others? This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept. How honest are the students when assessing each other’s work? 10/31

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 Goals: • To engage the students in practice and study • Without significant increase of the marking load 0. 25. 5. 75 1 Zero Poor 2 levels of fail Accept. Good Very good 3 levels of pass Problems: • Poor feedback • “Ethical” question: marking is based on This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept. comparison How honest are the students when assessing each other’s work? 11/31

(“Other” tests – 10%) Examples of “other” tests 1 0. 75 0. 5 This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept. How honest are the students when assessing each other’s work? 12/31

Description of a Case Study at IST Technical Drawing and Geometrical Modelling I & II Assessment “Other” tests CAD* assignment 10% In-class tests 30% Group project 40% *CAD – Computer Aided Design Group Public presentations 10% Lecturer 5% Peer-assessment 5% How honest are the students when assessing each other’s work? 13/31

Description of a Case Study at IST About the public presentation • Done in-class, during last week • 10~15 minutes/group • Students had to attend all the presentations • Marks ranged from 1 to 10 (0 for no shows) 1 -2 Very Poor 3 -4 Poor 5 -6 Accept. 7 -8 9 -10 Good Very good • Peer-assessment marks kept confidential by the lecturer • Non-differentiating peer-assessment marks were rejected How honest are the students when assessing each other’s work? 14/31

Outline 1. Why student’s peer-assessment? 2. Description of a case study at IST 3. Some of the students’ works 4. Peer-assessment results 5. Conclusions How honest are the students when assessing each other’s work? 15/31

Some of the students’ works Mower Boring mill Reactive suspension Rodrigo Ferreira Wilson Carreira Hélder Camacho Filipe Moreira Leonor Neto Soraia Ribeiro How honest are the students when assessing each other’s work? 16/31

Some of the students’ works A Racing bike!!! (Solidworks) (Author deliberately left anonymous) How honest are the students when assessing each other’s work? 17/31

Outline 1. Why student’s peer-assessment? 2. Description of a case study at IST 3. Some of the students’ works 4. Peer-assessment results 5. Conclusions How honest are the students when assessing each other’s work? 18/31

Peer-assessment results First observation: • Both the teacher’s and students’ marks follow a Gaussian distribution. How honest are the students when assessing each other’s work? 19/31

Peer-assessment results How honest are the students when assessing each other’s work? 20/31

Peer-assessment results The students did not fail their colleagues How honest are the students when assessing each other’s work? 21/31

Peer-assessment results The two students with very poor presentations are given a pass by their peers. . . How honest are the students when assessing each other’s work? 22/31

Peer-assessment results. . . and the four students just below pass were given a comfortable pass by their peers How honest are the students when assessing each other’s work? 23/31

Peer-assessment results 72% of the marks given by the students are in the 7 -8 (Good) range How honest are the students when assessing each other’s work? 24/31

Peer-assessment results However, the students seem to be afraid to overrate too much their colleagues How honest are the students when assessing each other’s work? 25/31

Peer-assessment results • The averaged values are not very different • The teacher’s standard deviation is practically twice as much as the peers’ How honest are the students when assessing each other’s work? 26/31

Peer-assessment results The marks are concentrated around the same mode, although the teacher’s assessment “leaks” towards the whole marking range. How honest are the students when assessing each other’s work? 27/31

Outline 1. Why student’s peer-assessment? 2. Description of a case study at IST 3. Some of the students’ works 4. Peer-assessment results 5. Conclusions How honest are the students when assessing each other’s work? 28/31

Conclusions Overall conclusions from this study: • Students have a tendency to overrate their “mates” • They are able to make a correct relative judgement between their peers’ performances (and thus themselves) • Most often, they will avoid to: • Fail their “mates” • Give their peers topper marks How honest are the students when assessing each other’s work? 29/31

Conclusions How honest are the students when assessing each other’s work? Qualitatively honest Quantitatively biased Students move by their own code of conduct and live in a competition environment. How honest are the students when assessing each other’s work? 30/31

Acknowledgments are due to Prof. António Ribeiro, Assistant Professor at IST, for sharing the data from two of their groups of students. THANK YOU Diogo Montalvão d. montalvao@herts. ac. uk Senior Lecturer, Dipl MSc Ph. D CEng School of Engineering & Technology How honest are the students when assessing each other’s work? 31/31