WA Health workplacebased assessment for international medical graduates
WA Health workplace-based assessment for international medical graduates 1
Australian Medical Council § Three pathways to registration: § Competent Authority § Specialist Pathway § Standard Pathway § Assesses the knowledge, clinical skills and professional attributes of international medical graduates (IMGs) on the standard pathway only seeking registration to practise medicine in Australia. 2
Australian Medical Council exams § English language exam § AMC 1 MCQ 180 questions § AMC 2 clinical exam MCAT/OSCE – Competency assessment – Role plays (pass = 12/16) – Pass rates in October 2016 were 35% (i. e. 106 of 305 IMGs passed) 3
AMC exam outcomes § After sitting the AMC clinical exam, IMGs historically demonstrate: § § § Performance issues Poor communication skills Inability to integrate well into a team Poor understanding of cultural awareness Poor clinical judgement was most significant concern § Outcome is less likelihood of retaining in WA Health 4
Miller’s Pyramid of Clinical Competence Further information: http: //winbev. pbworks. com/f/Assessment. pdf 5
Workplace-based Assessment Standard Pathway § AMC examinations § AMC MCQ CAT and the AMC clinical § Assesses at ‘knows’ and ‘knows how’ levels § Workplace-based Assessment (WBA) § AMC MCQ CAT and a program of WBA of clinical skills and knowledge by an AMC accredited authority § Assesses at the ‘does’ level § Focuses on performance in the workplace setting § Provides opportunities for immediate constructive feedback 6
Workplace-based Assessment § Introduced under the 2007 COAG Nationally consistent assessment of IMGs initiative. § Established to assist with: § Workforce issues, particularly in rural and regional areas § The large number of IMGs waiting to sit the AMC clinical examination. § WBA has been adopted at 15 hospitals nationwide across 7 health providers. § WBA programs are accredited in accordance with AMC WBA Guidelines and Criteria while taking into consideration local circumstances. 7
WBA opportunities § Attraction and retention of clinical staff in the many WACHS sites continues to be a challenge. § Upsurge in Australian-trained medical graduates but continued reliance on IMGs for service delivery. § WBA has an significant impact on interest in employment at participating WACHS hospitals. § There approximately 20 candidates every year. § Of those remaining in WA after obtaining general registration, on average, 70% continue to work in rural areas. 8
WBA benefits § Rigorous with a much higher pass rate than the AMC clinical. § Supports integration into the local community and assimilation into the Australian health workforce. § Assessment and feedback methods provide consistency of supervision and assessment techniques within the wider hospital community. 9
What is WBA § A form of authentic assessment, testing performance in everyday clinical practice involving the measurement of abilities and attitudes within highly complex areas of activity. • Must have established reliability and validity to ensure that defensible decisions are made through using: § A variety of assessment methods § Multiple observations in various clinical settings § Trained multiple assessors 10
What is WBA § Delivered in all states and well supported. § Involves observation, assessment and immediate feedback of candidates in the workplace. § Global ratings developed. § Longitudinal assessment over 12 months. § Can track development towards independence. § Improves clinical skills, communication and teamwork. § Identifies under-performance and has action plans to support improvement. 11
WBA in WA § The WA Department of Health is the AMC accredited provider for WBA. § Hospitals implementing WBA must be AMC accredited. § Currently accredited sites: Bunbury and Geraldton Hospitals and Kalgoorlie Health Campus § Accredited assessment plan spans over 9 months. § The length of the WA WBA program aligns with the Medical Board of Australia requirements for eligibility for general registration. 12
Entry to the program § Complete AMC MCQ, English exam and Electronic Portfolio of International Credentials (EPIC) verification § Must be eligible for limited registration on the standard pathway with the Medical Board of Australia § Employment with WACHS (contract to be provided) 13
Assessment? Assessment OF learning……… OR Assessment FOR learning……. . 14
Assessment level “an Australian intern at the end of the PGY 1 year” or “that which one would expect from a minimally or just competent medical officer at the end of PGY 1” Further information Intern training – intern outcome statements Australian Curriculum Framework for Junior Doctors 15
Assessment goal • A sampling of a candidate’s performance. • Must be representative of the spectrum of that relevant to safe medical practice. • Assessors need to be sufficiently familiar with the patient to enable them to critically judge the performance being reviewed. 16
Assessment process • • • Part of everyday clinical practice. Candidate is responsible for own learning. 6 clinical areas and 6 assessment tools. 12 month program. Must pass a set number of tools (candidate blueprint). Must have minimum 10 hours of exposure to a clinical area (i. e. , ward rounds, clinics etc) before being assessed against clinical skills and dimensions. 17
What is an assessor? Clinicians who are: § Experienced, and possess expertise in the clinical tasks assessed. § Trained in WBA for target cohort (IMGs). § Available. § Have agreed to assess according to the principles of WBA. 18
What can an assessor do? § Select the patient/case and assure consent is received. § Ensure the candidate has undertaken adequate preparation. § Conduct direct or indirect assessment in a clinical area and provide immediate feedback. § Undertake a formative assessment and provide feedback if requested. 19
Direct assessment rules General or specialist registrants who have successfully completed 4 years of experience in the Australian health care environment. OR Equivalent experience trained in a designated Competent Authority country. • DOPS may also be assessed by registered nurses with appropriate clinical assessment experience. Other AMC candidates are not to be included as assessors or patients 20
Indirect assessment rules • The WBA provider should have clear statements of the expertise and experience required for the appointment of assessors. • All assessors must have: • Completed the WBA assessor and supervisor training package. • Signed the declaration that they have completed this training. • In the case of MSF, the candidate and/or WBA provider may choose to include other members of the health care team. 21
Conflict of interest • AMC and WA Health take this issue very seriously • The WA Health Employment Policy Framework applies to the all WA Health staff. • Specifies employment governance requirements that apply across the WA health system. Further information http: //www. health. wa. gov. au/Circulars. New/Employment. cfm 22
Assessment areas • Must complete assessments across the 6 clinical areas covered in the AMC clinical exam § Adult health (medicine) § Emergency medicine § Adult health (surgery) § Child health § Mental health § Women’s health • Child health and women’s health are not traditionally assessed at intern level. • Mental health has been identified as a clinical area that some candidates may not be familiar with. 23
Assessment blueprint • Assessment drives learning • Key is to assess ‘important things’ to learn and do. • A ‘blueprint’ should guide selection of encounters: • Problems from different systems/clinical areas • Clinical tasks (e. g. history taking, examination) • Clinical settings (e. g. emergency departments, clinics) within which observations will occur. 24
Blueprint - macro 25
Blueprint - micro 26
Assessment tools • • 12 mini-clinical examinations (Mini-CEX) 6 direct observation of procedures (DOPS) 6 case based discussions (CBDs) 3 in-training assessments (ITAs) 1 multi-source feedback (3600) 1 external assessment Additional assessments § 1 Self-assessment and Learning Plan § 2 formative ITAs § 5 candidate self-assessments 27
WA Health approved assessment plan Direct assessment methods • 12 Mini-CEX – 2 in each clinical area (or more as needed) • 6 DOPS – 1 in each clinical area (or more as needed) • 1 external assessors report – direct observation (calibration tool) Indirect assessment methods • • 6 CBD – 1 in each clinical area (or more as needed) 2 Formative supervisor’s reports. Between 18 -20 weeks apart 2 Summative supervisor’s Between 18 -20 weeks apart. MSF – 10 colleague evaluations usually undertaken mid-year. Collated into a report for the primary supervisor to discuss with and provide feedback to the candidate. 28
Mini-CEX • Process of directly observing a candidate in a focused patient encounter for purposes of assessment, followed by feedback on performance. • Observation is typically 10 -15 minutes on a focused task. • 10 -15 minute highly focussed feedback session. • Not all aspects of clinical encounter covered with every patient. • A recent study found that scores derived from as few as 10 mini-CEX encounters possessed a reliability coefficient exceeding 0. 80. 29
Assessors role in Mini-CEX • Totally uninvolved in the encounter and as unobtrusive as possible, unless there are risks to patient safety. • Any issues identified should be followed up after the candidate has completed the encounter with the patient. • All questions on the Mini-CEX form should be completed with both effective and ineffective aspects of performance noted. • The Mini-CEX forms should be returned to the WBA administrative officer for data entry and record keeping. 30
Mini-CEX Examples of end of PGY 1 characteristics Medical interviewing and communication skills • Uses questions effectively to obtain an accurate, adequate history with necessary information, and responds appropriately to verbal and non-verbal cues Organisation/ efficiency • Makes efficient use of time and resources Physical examination skills • Follows an efficient and logical sequence Professional/ humanistic qualities • • Is aware of safety issues Maintains a professional approach to patient Counselling skills • Demonstrates an understanding of different cultural beliefs, values and priorities regarding their health and health care provision, and communicates effectively Clinical judgement • Integrates and interprets findings from the history and/or examination to arrive at an initial assessment, including a relevant differential diagnosis Overall clinical competence • The assessor makes a global judgement based on the whole encounter and considering performance at end of PGY 1 level 31
AMC feedback • All boxes/fields must be completed on the assessment forms, including comments in all areas. • Forms must be signed by candidate and assessor • When a task or requirement is not observed the assessor must provide a reason. • DOPS forms must indicate what is being assessed (i. e. , procedure or information) and must include comments from the assessor. • The quality of assessments must be readable. • Suggested minimum observation and feedback times of 10 -15 minutes for each. 32
Resources WBA Online http: //wbaonline. amc. org. au/ The assessment of clinical skills/competence/performance http: //winbev. pbworks. com/f/Assessment. pdf WA Health international medical graduate website http: //ww 2. health. wa. gov. au/Careers/Internationalapplicants/International-medical-graduates-IMGs 33
- Slides: 33