REMOTE MARKERS SUMMER 2017 Standardisation Briefing Units Subject

REMOTE MARKERS SUMMER 2017 Standardisation Briefing Units Subject to Batch Checking

Key Contacts Type of Query Whom to Inform Contact Number/Email Address Assessment Queries Team Leader Details will be provided to you by your Team Leader Script Allocations Assessor Management As. deployment@ocr. org. uk Withdrawing from Marking Team Leader and Assessor Management As. deployment@ocr. org. uk Technical issues with RM Assessor RM Helpdesk 03303 336 319 Please indicate unit and examiner creditor number as well as nature of query. Please indicate unit and examiner creditor number. Also inform us whether your withdrawal is for this session only or permanently. Administrative Queries On Screen Marking Helpdesk Related to your marking 01223 553393 Instructions and Forms http: //www. ocr. org. uk/assessorcommunications/ Assessor Communications website

Regulatory requirements Examining is regulated by condition H 1. 1 of the General Conditions of Recognition. This requires that the awarding organisation must • have in place effective arrangements to ensure that, as far as possible, the criteria against which Learners’ performance will be differentiated are – (a) understood by Assessors and accurately applied, and (b) applied consistently by Assessors, regardless of the identity of the Assessor, Learner or Centre.

Purpose of Standardisation The standardisation process is designed ensure that all examiners have a well-founded and common understanding of the requirements of the mark scheme (appropriate to their responsibilities) and can apply them reliably. Completing this process enables all examiners to mark candidates' work consistently and accurately. It establishes a common standard of marking that should be used to maintain the quality of marking during the marking period.

Responsibilities • The Principal Examiner is responsible for the professional judgements underpinning the process of standardisation. • Team Leaders are responsible for supporting examiners through the standardisation process.

Standardisation for Remote Markers(1) • Question Paper can be accessed from the RM Portal day after paper sat date. • SSU (standardisation set-up meeting) take places 7 -10 days after paper sat date (this is attended by Team Leaders and the Principal Examiner). • Mark scheme is finalised at SSU and practice and standardisation scripts are selected for the learning and testing phases for remote markers. RM Portal http: //support. rm. com/generic. asp? Cref=GP 1286643&Src. URL=/ca

Standardisation for Remote Markers(2) • After the SSU your Team Leader will make contact with you to inform you that the mark scheme will be available via RM Assessor(this should be within 24 hrs of the end of the SSU). • Once you have accessed the mark scheme you should login to RM Assessor and access the practice scripts to begin the learning phase, followed by the testing phase. • This process is likely to commence approximately 10 -12 days after paper sat date (timescales can be shorter for later examination papers).

Standardisation for Remote Markers(3) Exam is Sat Marker accesses the Question Paper SSU Meeting (Approx 7 -10 days Paper Sat Date ) After SSU - TL contacts team Marker can now access mark scheme from RM Assessor Marker can now begin the process

RM Assessor All marking will be carried out on RM Assessor. URL https: //ca-marking. assessor. rm. com

Standardisation Approval Process (1) • Download practice scripts, then mark and submit one at a time. Once you have submitted the first script, view the definitive mark by accessing the script in your closed items before continuing with the next practice script. • Email your Team Leader once you have finished the practice scripts. Your Team Leader will then contact you and provide feedback. This is your opportunity to ask any questions about how the definitive marks were awarded. • After this discussion has taken place, download the standardisation scripts. This is your test of marking. Mark these scripts independently and submit all at once. • Your Team Leader will then review your scripts and either: • - approve you to complete live marking or • - not approve you and provide feedback. You will then be asked to complete a second test (STM scripts) which you will need to download, complete and submit to your Team Leader.

Standardisation Approval Process (2) • After second test (if applicable) TL either approves marker for live marking or informs the marker and the examiner and assessors team that the examiner will not be progressing to live marking. Guide Location Completing Standardisation Process On Screen Marking, RM Assessor Examiner/Specialist Marker, Quick Prompt Marker guide On Screen Marking, RM Assessor, Marker Guide RM Assessor Documents can be located at http: //www. ocr. org. uk/assessor-communications/

Standardisation Approval Process (3) Familiarisation and Learning Practice Scripts Testing Standardisation Scripts Approval Live Marking Non Approval Second Test STM Scripts Non Approval Examiner does not progress to Live Marking TL informs Examiner and Assessor Management

Suspension during live marking (1) • If a Team Leader has concerns regarding your marking (as a result of your marking on seeded scripts) they will check some live scripts. • If following the live marking check your Team Leader still has concerns they will suspend you on the system and contact you to inform you that you are suspended from any further marking until the PE has reviewed the marking. • The PE will then review the judgements made by your Team Leader (as soon as possible) and then inform OCR and your Team Leader of the outcome. • This could result in the following: – Your Team Leader makes contact with you to inform you that you can continue marking or – An OCR Monitoring and Support Manager contacting you to inform you that you will not be able to mark any further scripts.

Suspension during live marking (2) TL carries out check Completes live script checking form Check reveals concerns TL Suspends Marker Sends completed form to Monitoring and Support Manager and PE Examiner must be stopped Monitoring and Support Manager contacts Examiner to inform them of decision PE considers Evidence (24 hrs) Confirms Decision to Monitoring and Support Manager/TL Examiner should continue marking TL informs marker and removes suspension

Monitoring (1) Batch Checking • This is the main mechanism for monitoring on this unit • At four points in the marking period, your supervisor(TL, STL, PE) will look at your live scripts in order to see if there any patterns in your marking. • At each (or batch), your supervisor will select no fewer than 5 scripts and second mark these, recording the overall script totals (TL versus AE) on the Batch C heck Form. • The supervisor will select these scripts being mindful of sampling a reasonable mark range and question options You will not receive any feedback except on annotations.

Monitoring (2) Seeds • Seeds are used as a quick way to check marking accuracy on a regular basis. • Again, you will not receive feedback except on matters of annotation • We want markers to maintain the same standard of marking (whether it is slightly generous or slightly severe or spot on), rather than overcompensating following feedback.

Marker Monitoring • All examiners are monitored through seeding scripts and our marker monitoring alert system. • Where an examiner’s marking is not in line with the rest of the markers on the panel or their response to a seeding script is causing concern an alert is triggered. • Once the alert is triggered the Team Leader is asked to carry out a check on the live marking of the examiner the alert has been triggered on. • Following the check on live marking the team leader either informs the monitoring team the examiner is fine to continue or if the Team Leader has concerns then they will suspend the marker while the PE considers the evidence.

TL is requested to carry out a check This request is made by the Monitoring and Support Team TL suspends marker on Scoris and advises them of suspension TL carries out check (48 hrs) Completes Alert Response Form (ARF) Check reveals no concerns Sends completed form to Monitoring and Support Team Check reveals concerns Suspends Marker If a marker is to be stopped from marking an OCR Monitoring and Support Manager will make contact. Examiner must be stopped Monitoring and Support Manager Contacts Examiner Confirms to PE/TL Examiner has been stopped Sends completed ARF to Monitoring and Support Manager and PE considers Evidence (24 hrs) Confirms Decision to Monitoring and Support Manager/TL Examiner should continue Marking TL informs marker and removes suspension

Mark Return Dates and Grace Period Mark Return Dates • Marks return dates are available on the Assessor Communications website and will also be available to you within RM Assessor once you start marking Grace Period • The marking grace period starts at 48 hours. • It reduces to 12 hours after the 50% marks return date. • Reduces to 0 hours after the 100% marks return date. Assessor Comms Website (under the heading forms): http: //www. ocr. org. uk/assessor-communications/

Confidentiality of Marking • Marking must never be carried out in a public place. • There must be no discussion of work in progress with anyone other than members of the marking panel and OCR. Therefore, do not discuss marking with anyone else including other colleagues or your family. You should never discuss any aspect of marking or marking process with the media.

Conflict of Interest (1) What to do if you download a response and you think you know the candidate Although it is very unlikely, there is a remote possibility that you may recognise that a response you have downloaded for marking has been written by a candidate who is known to you. This is a conflict of interest, so if you recognise a response that you have downloaded, you must reject this immediately. Where a response you have downloaded for marking has additional objects, you are advised to check these for any identifying details before you start marking it, so that you can reject it immediately if you recognise the candidate.

Conflict of Interest (2) To reject a response: 1. From the Marking page for the response, click Reject Response. A confirmation request is displayed. 2. Click Yes. The response is rejected and the next available response is displayed.

Conflict of Interest (3) Please note: this method of rejecting scripts should only be used where an examiner recognises a candidate response. For instances such as illegible scripts or where responses come from an unfamiliar option route, the appropriate exception process should be raised.

Marking – Good Practice • It is important that you mark at a steady rate to ensure that you are not rushing to complete your marking to meet the deadlines set. • It is important to let you team leader know if you are planning to take any long breaks during the marking period e. g. not marking any scripts for a couple of days

Annotation Purpose and Importance of Annotation • Provides an audit trail of why marks have been awarded. • Improves visibility of marking process for centres. • Allows TLs/PEs to look at AE scripts and understand where/how marks were awarded. This means useful feedback to AEs can be given. Mark scheme • Follow annotation principles as laid down in the mark scheme. • There must be evidence that every page has been seen by an assessor, including blank pages. Guidance will be provided in the mark scheme.

Giving Marks (1) Implications of incorrectly entered marks • Incorrectly entered marks (e. g. through mis-keying) result in candidates being issued with the wrong grade. This brings the integrity of the examination into question. Ensure that you: • activate on screen mark confirmation so that when you enter a mark a large image of the number you have entered will appear briefly on screen. • are marking against the right image zone (for optional questions).

Giving Marks (2) PLEASE READ CAREFULLY INFORMATION BELOW -THERE HAS BEEN A CHANGE TO THE GUIDANCE ISSUED IN PREVIOUS SESSIONS. Award No Response (NR) if: • There is nothing written in the answer space Award Zero ‘ 0’ if: • anything is written in the answer space and is not worthy of credit (this includes text and symbols). Team Leaders will confirm the correct use of the NR button with their markers before live marking commences and should check this when reviewing scripts.

Rubric Errors PLEASE NOTE • Where candidates have a choice of questions and have provided more answers than required, then all responses are marked and the highest mark allowable within the rubric is given (by the system). • Enter a mark for each question answered into RM assessor, which will select the highest mark from those awarded. This slide is only applicable to papers were there is optionality

Appraisal & Grading of Examiners At the end of the marking period your Team Leader will be asked to grade you on the following: • Quality of Marking • Administration/Communication This grading will determine whether or not you require any further training before you are invited to mark for future sessions.

Payment for Marking • Payment will only be made for scripts that have been marked and submitted. • Payment for marking will be made automatically at regular intervals during the marking period for scripts that you have marked and submitted.

Finally…………. Thank you for the marking that you will be undertaking for OCR for the June 2017 session.