Behind the scenes at SQA How has quality

Behind the scenes at SQA How has quality assurance evolved for exams? SQA Coordinator Events – May / June 2017

Aim of today’s seminar A look at how the marking and quality assurance of exam scripts has changed over time: w w w The ‘traditional’ marking method Quality assurance of traditional marking Today’s e-marking process Benefits of e-marking E-marking quality assurance Future plans

‘Traditional’ marking w Pen and paper w At home w Scripts allocated to markers in advance w Administration and postage

How were markers trained? w w Attend a markers’ meeting Led by PA and exam team members Administrative points covered Understanding of national standards o how to apply the marking instructions o anonymous scripts used to exemplify features of the marking instructions

The traditional marking period w Around 3 weeks to mark w In packets of centres w No direct link to the exam team – marking issues returned to PA w If unable to complete allocation –scripts returned to SQA to be re-allocated to another marker

Traditional quality assurance: checking the marking w After marking complete (‘end loaded’) w Exam team sample scripts w Each marker’s allocation is sampled: o Marking to national standard? o Any deviation from national standard – severity, leniency or inconsistency? o What action is necessary?

Traditional quality assurance: finalising marks w After grade boundary setting and marker sampling w Scripts prioritised w Close to grade boundary? w Close to the C boundary? w Severely marked? w Leniently marked? w Does school estimate support change in grade?

Re-marks w Significant marking issues found during sampling require different action w Re-marks - full allocation or particular questions w Decision taken in consultation with various business teams and the PA w Operations staff are key in planning and facilitating

So why change? An introduction to e-marking

What is e-marking? w Candidate scripts sent to central location for scanning w Images made available for markers w Images are anonymous w Suitable for all types of assessments

E-marking progress 2012 2013 2014 2015 2016 2017 6% all exam scripts e marked 17% all exam scripts e marked 55% all exam scripts e marked 75 % all exam scripts e marked 98 % all exam scripts e marked

E-marking quality assurance model Standardisation and preparation for Marker briefing Markers are briefed and introduced to Team Leaders Markers will complete a series of Practice scripts Markers must pass a set number of Qualification scripts Live marking Continuous sampling using seeded scripts and manual sampling/monitoring by Team Leaders

What are the benefits of e-marking? Assessment and Quality Assurance: • Confidence in assessment decisions • Upfront quality assurance • Item level analysis

What are the benefits of e-marking? Other benefits • Eliminates clerical errors • Responses are randomly allocated • Personal details removed from responses • Helping the environment

What are the benefits of e-marking? Support and training • Ongoing support and increased marker satisfaction • Further training opportunities – Understanding Standards programme • Markers sharing standards with colleagues

SQA’s future approach w Scottish Government’s Digital Learning and Teaching Strategy, Sept 2016 includes our commitment that: ‘SQA will, over time, develop the capabilities needed to manage assessment evidence in a digital format for all of its qualifications. w SQA continues to investigate a range of digital approaches and opportunities to use technology in our qualifications. w Looking further ahead, can paper responses be supplemented with submission via laptop? SQA is starting to investigate the challenges that this would involve.