The Automatic Generation of Marks for Marking within

The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays Phil Davies School of Computing University of Glamorgan “Super U” 1

Computerised Peer-Assessment § CAP System § Permits students to mark & comment the work of other students. § Also self-assess … used as a standard of expectation § Internet, not Web-based system (developed in Visual Basic / Access) 2

3

Having done the marking, what next? § Students should receive feedback § What feedback? – Marks – Comments § Which is most important? 4

5

AUTOMATICALLY EMAIL THE MARKER. . ANONYMOUS 6

What should the marker do? Reflect § Look at essay again § Take into account the essay owner’s comments § Further clarification (if it is needed, then is this a ‘black mark’ against the marker? ) § Try to ‘appease’ the essay owner? § Modify mark based upon reflection? § Give more feedback 7

8

Must be rewarded for doing the ‘mark for marking’ process § How to judge? § § § Standard of expectation (self-assessment) Marking consistency Commenting, quality, measure against mark Discussion Element Need for additional comments – black mark? Reaction to requests / further clarification 9

Does this student deserve a good mark? ESSAY Student Mark 67% Difference W Compensated Median Mark? ? 65% X 48% 54% +6% Y 53% 57% +4% Z 61% 59% -2% +2% on average 10

Standard of Expectation Self-Assess = 68% Peer-Assessed as = 58% ESSAY Student Mark 67% Difference W Compensated Median Mark 65% X 48% 54% +6% (4) Y 53% 57% +4% (6) Z 61% 59% -2% (12) X +2% (7. 5) +2% (8) 11

12

How easy is it to do? § Statistically fairly easy to create a mark for marking based upon marks § Take into account high and low markers § Standard of expectation § Consistency … judge against final mark awarded for an essay (compensated median) § What about the comments? 13

Feedback Index § Produce an index that reflects the quality of commenting § Produce an average feedback index for an essay § Compare against marker in a similar manner to marks analysis § Where does this feedback index come from and is it valid? 14

The way to get the feedback index? § Develop an application? ? – C-Rater? § Spelling Mistakes § Similar Meanings? – That was cool – Really Choc – Really Good Essay § Manually 15

16

17

18

19

20

21

22

23

Differences from actual: F/Index DAI SMITH Mark Score ESSAY 1 4. 5 3. 8 3/10 ESSAY 2 -3. 0 8. 5 5/10 …………. … FINAL MARK AWARDED 5/10 24

Time Consuming? § Can we formulate the marking process § Take away need for quantification process of analyzing comments § Is it still peer-assessment if the students are told what to say? 25

26

STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2 27

Some points outstanding § What should students do if they identify plagiarism? § Is it ethical to get students to mark the work of their peers? § Is a computerised solution valid for all? § At what age / level can we trust the use of peer assessment? § How do we assess the time required to perform the marking task? 28

The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays § § Student Acceptance: Anonymity / Discussion / High & Low Markers Tangible Reward I’m more concerned with rewarding ‘higher order skills’ § Getting there 29