WorkplaceBased Assessment Making it More Effective Jonathan Beard

Workplace-Based Assessment Making it More Effective Jonathan Beard Professor of Surgical Education University of Sheffield, UK

Objectives • Check knowledge of Workplace Based Assessment (WBA) • Understand the purpose of assessment • Review some of the problems of WBA • Discuss possible solutions • Any other objectives?

Workplace Based Assessment “Assessment of day-to-day working practices undertaken in the working environment” • WPBA now an integral part of medical training (competence-based curricula) • Integrates teaching, learning, assessment & feedback • Designed to assess skills rather than knowledge • Competence best assessed in the workplace as closest to actual performance • Primary aim is to aid learning

WBA Tools • Mini-CEX • DOPS • Specialty tools Opportunistic – e. g. PBA (ISCP), ACAT (RCP) • CBD Planned • Assessment of Teaching or Audit • MSF: Once per placement – Only tool that does not require rater training

What is the Point of Scoring WBA? It depends on the purpose of the assessment

Purpose of Assessment? • To aid learning through constructive feedback: – Assessment for Learning (formative) – Done frequently • To check knowledge or skill has been learned: – Assessment of Learning (summative) – Done infrequently

Time for Teaching and Learning How many hours of teaching and learning does a trainee get per week?

Learning to Drive Driving Lesson Assessment for Learning Driving Test Assessment of Learning

The Danger of Confusing the Purpose of WBA • Doctors regard all assessments as tests • Therefore they practice informally before requesting an assessment, choose easy tasks and make the request after the event • Assessors feel pressured to give a good score • More attention paid to score than feedback • Primary purpose of WBA as an aid to learning is lost

Changing the Name • Supervised Learning Events (SLEs) done frequently to aid progress – trainee chooses the topics and timing – No score but feedback must be documented • Assessments of Performance (Ao. Ps) done less frequently to check that key competencies have been achieved – Identified by the specialty curriculum – ‘Satisfactory’ or ‘development required’

Frequency and Timing of WBA Level of Competence Able to perform the task Supervised Learning Event Assessment of Performance Placement One Placement Two Training Rotation Placement Three

Changing the Language • Similar format for many existing assessments • Norm-based rating scale: 16 where 4 = ‘meets expectations for completion of stage of training’ • Standard not defined but can usually be found ‘somewhere’ in the syllabus

What does ‘Meets Expectations for Completion of a Stage of Training’ Mean? • • • To the Trainee? To the Assessor? To the ES/TPD? To the Employer? To the Patient?

Evaluation of WBA in Surgery • DOPS – 1370 trainees (ST 1 -4), 6240 DOPS (m=3. 4) – Most domains rated 4 or 5 (m=4. 8) – No construct validity and no reliability • Similar findings by RCP for CEX and CBD • PBA (Procedure-Based Assessment) – Criterion-referenced rating scale – 85 trainees (ST 1 -7), 749 PBAs – Good internal and construct validity – High reliability (G>0. 8 for 3 assessments)

Redesign of WBA Tools • Criterion-referenced with clear performance anchors (e. g. could perform the task without help) • Use descriptors not scores (e. g. able to do the task) • Emphasis on feedback (e. g. make documentation of feedback mandatory) • Clearer guidance notes for trainees (e. g. timing and frequency) and assessors (e. g. seniority and training)

Revised Surgical DOPS (ISCP) Verbal feedback and documentation of feedback is mandatory

Revised Ratings for CBD and Mini-CEX (RCP)

Assessment Methods • Can the same assessment method be used for both purposes? • Yes - but only if its use on each occasion is agreed and documented • And - feedback is documented if it is an SLE (remove all ratings? ) • And - reliability is high enough for an Ao. P (well trained assessors)

Example of Implementation for Surgical Specialties • 40 SLEs per year (1 per week) • 3 Ao. Ps (PBAs rated at level 4) required for every index procedure before CCT • Logbook will ask whether an SLE or Ao. P was done • SLE feedback must documented by trainee and validated by assessor • Ao. Ps can only be done by CS or ES

Summary • Postgraduate medical curricula will need to be amended in light of the GMC discussion document • No sudden change required – direction of travel – but need to be considered when reviewing curricula • Trainers and trainees need an update • Don’t mention the ‘A’ word • Ask the question: driving lesson or test?

Further Reading and Information • Further reading: – Impact of workplace based assessment on doctors’ education and performance: a systematic review. Miller & Archer, BMJ 2010 – Changing the Name and the Language. Crossley and Jolly. Medical Education 2012 • More information: – http: //www. gmc-uk. org/Standards_for_Curricula__Assessment_Systems. pdf_31300458. pdf – http: //www. gmc-uk. org/Workplace_based_assessment_31381027. pdf – http: //www. gmcuk. org/Learning_and_assessment_in_the_clinical_environment. pdf_45877621. pdf

Some Questions • Do we need separately designed forms for SLEs and Ao. Ps? • How many hours of teaching and learning should a trainee have per week? • Should MSF be designated as an Ao. P? • Should Ao. Ps only be done if there is no better way of testing a skill/behaviour? • How do we communicate the new terminology and philosophy to trainers and trainees?