Laboratory Exams in First Programming Classes Quintin Cutts

Laboratory Exams in First Programming Classes Quintin Cutts, University of Glasgow quintin@dcs. gla. ac. uk and the Disciplinary Commoners Dave Barnes, Kent; Pete Bibby, Bolton; Jim Bown, Abertay; Vicky Bush, Gloucestershire; Phil Campbell, LSBU; Sally Fincher, Kent; Stephan Jamieson, Durham; Tony Jenkins, Leeds; Michael Jones, Bournemouth; Dimitar Kazakov, York; Thomas Lancaster, UCE; Mark Ratcliffe, Aberystwyth; Monika Seisenberger, Swansea; Dermot Shinners-Kennedy, Limerick; Carol Wagstaff, Teesside; Linda White, Sunderland; Chris Whyley, Swansea. http: //www. cs. kent. ac. uk/~saf/dc HEA-ICS 7 th Annual Conf. 31 -08 -06

Outline • • • Motivation for laboratory exams Seven Commons models Comparing the models HEA-ICS 7 th Annual Conf. 31 -08 -06

Open assessed coursework • Solution created by – – – student another student last year’s student book author web author contract coder • Ensure ‘student’ by – – – trust their good nature warn of severe penalties manual code search automatic code search horrendous institutional disciplinary route – asking questions about understanding (almost) HEA-ICS 7 th Annual Conf. 31 -08 -06

Coursework in an ITP course… • Primary skill development mechanism • Should skill development be mixed with skill assessment anyway? – No!! Increased stress etc. • Hence – use a more formalised assessment of programming skills HEA-ICS 7 th Annual Conf. 31 -08 -06

define: laboratory exam • • Strictly controlled set of rules Possibilities – – time limit unseen questions limited access to other materials limited access to other people • Rules upheld by strict invigilation HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Max students exam tried with No. of examinations in course No. of sittings required for each exam No. of versions of paper required for each exam Length of exam Is it an open book exam? Is the paper/question seen in advance? Can the student work with/confer with others? Are the laboratory machines networked? HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Aber Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd 120 4/yr 1 Smaller tail and std dev End of course MCQ answered much better 1 120 y n y Subm & marking on-line maximises feedback Tests comprehension not memory y HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Aber UCE Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd 120 160 4/yr 2/smr 1 2 smdy 1 6 120 90 y y Provision of early f’back on progress n n 2 nd test based on c’work y y Security paramount y y 1 st year of use Low ave. mark, but many students absent this time HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Aber UCE Dur 120 160 80 4/yr 2/smr 2/yr 1 2 smdy 5 1 6 1 sc+5 120 90 110 y y y n n n y y tut avl y y y-m/e 10% used help facility, reduced stress (no tears) Allows students stuck at one point to succeed elsewhere HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Aber UCE Dur Gla 120 160 80 4/yr 2/smr 2/yr 1 2 smdy 5 1 6 1 sc+5 1 120 90 110 y y y n n y y limited One version of exam only for multiple n y sessions tut avl y y-m/e 24 Some rote learning Assesses coding, testing & debugging only 110 n HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Aber UCE Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Dur Gla Lee 120 160 80 480 Results match performance on 4/yr 2/smr 2/yr other assessments 1 2 smdy 5 24 200 1 Matches 6 intended 1 sc+5 learning 1 outcomes 120 90 with assessment 110 1 1/crs 1 ~180 y y y limited y n n n y y tut avl y y y-m/e limited n HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Aber UCE Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Dur Gla Lee Sun 120 160 80 480 Students appreciate the 4/yr 2/yr validity 2/smr of the exam 200 1/crs 3/yr 1 Rapid feedback 2 smdy 5 1 4 1 6 sc+3 ~180 75 24 1 Heavy 6 to set up, 1 sc+5 1 but scenario reuse ameliorates 120 90 110 y y y limited y y n n n y y tut avl y y y-m/e limited y n HEA-ICS 7 th Annual Conf. 31 -08 -06

Vital Statistics Studs Freq Sess Vers Len Open? Seen? Indivi Ntwkd Aber UCE Dur Gla Lee Sun Swa 120 80 480 200 100 5 1/crs 3/yr 1 rsit 1 4 1 1 6 sc+3 1 160 4/yr 2/smr 2/yr The reality of a practical exam 1 2 smdy 5 24 wakes some students up to 1 necessary 6 study 1 sc+5 levels 1 120 110 ~180 Hence, 90 significant motivational yeffect y y limited y 75 120 y y n n n y y tut avl y y y y-m/e limited y y n HEA-ICS 7 th Annual Conf. 31 -08 -06

Three categories • Single session, single unseen paper – Aberystwyth, Leeds, Swansea • Multiple sessions, multiple unseen papers – UCE, Durham, Sunderland – Maintains fairness and reliability • Multiple sessions, single seen paper – Glasgow – Reduced assessment coverage HEA-ICS 7 th Annual Conf. 31 -08 -06

Reliability • No cheating/communication – invigilators – limited/no network – different versions of exams • Same complexity across all versions? – range of problems, all using same concepts – single overarching scenario, multiple problems drawn from it HEA-ICS 7 th Annual Conf. 31 -08 -06

Validity • Does assessment match real world activity? • access to books/web/earlier solutions good • access to human assistance (at a price) • Do assessment exercises match skill development exercises? HEA-ICS 7 th Annual Conf. 31 -08 -06

Scalability • Large class sizes not a problem now, but… – we can hope • Not all these models scale HEA-ICS 7 th Annual Conf. 31 -08 -06

Conclusions • Support for lab exams – – – Improved attitudes and study habits Reliable feedback on progress Assessing full range of programming skills Testing comprehension not memory Separating skill development from skill assessment Reducing exam stress • Further work – Analysis of the problems used – How were the students’ solutions evaluated? – Comparison of exam performance against other assessments • Disciplinary Commons HEA-ICS 7 th Annual Conf. 31 -08 -06