Computerised Freetext Assessment of Key Feature Problems Adrian

Computerised Free-text Assessment of Key Feature Problems Adrian Molyneux Steve Capey School of Medicine

Project aims n n Fulfil student requests for more formative assessment Fulfil need for practice of summative exam formats Provide quality feedback Achieve this without a heavy marking load placed on staff

Key-feature problems (KFPs) …an innovative assessment format that tests clinical decision making skills by focussing only on the critical steps - or 'key features' - of each decision, thereby increasing the number of problems that can be answered in an examination. - Liz Farmer, Royal Australian College of GPs

SCENARIO - Jennifer Evans, a 38 -year-old statistician, presents to the accident and emergency department with a two hour history of sudden onset of leftsided pleuritic chest pain shortness of breath and an episode of haemoptysis. She had returned from a conference in Malaga on a flight 48 hours earlier. On examination she was found to be slightly febrile, tachycardic and in atrial fibrillation. Her blood pressure is 98/64 mm. Hg. Examination of the chest showed decreased expansion on the left and localised decreasing breath sounds with a pleural rub. A plain radiograph of the chest shows a very small trace of fluid in the left costophrenic angle to the lungs are clear. A presumptive diagnosis of pulmonary embolism is made.

Example questions from scenario 1. What initial investigations should be requested? Name only two (2) 2. Long distance travel is considered a minor risk factor for venous thromboembolism – what are the major risk factors? Name only four (4) 3. What is the mechanism of action of heparin? 4. Shortly after the heparin was commenced, Jennifer was started on warfarin. What endogenous compound does warfarin inhibit to produce its anticoagulant effect? List only one (1) 5. How is warfarin therapy monitored?

Research questions n n n How do we train a computer system to mark an open-ended freetext response? How to deliver the assessment How to feed back the results to students How to author succinct KFPs Can we develop a method for the most efficient creation of a computer-marked KFP?

Methods n n n n Generate 10 KFPs and marking guidelines Generate computer markscheme Build web interface Recruit students and run exam Computer-mark responses Tutor-mark responses Refine markschemes (iterative process) Statistical analysis

Exam delivery system and results Exam delivery Results

Computer – Tutor comparison

Markscheme improvement n n Make first attempt at markscheme and run exam Examine student answers for further valid responses Include these in computer markscheme and mark a second time Examine computer markscheme for badly-performing questions and adjust (maybe loosen). Mark a third time.

Run Number Mean Mark % of Human Mark 1 0. 9 39% 2 1. 1 47% 3 2. 0 86%

Conclusions – practical limitations n n n Succinct answers should be encouraged – perhaps limit text entry Expect unexpected answers from students and markscheme modification Standardisation of human marking n n Adherence of humans to marking guidelines Evolution of computer markschemes to cover the whole range of student answers

Conclusions - positives n n n Computer marking ideal for questions with narrow range of possible responses Reduce marking hours spent on formative questions, whilst providing quality feedback in a realistic examination format Perfect for low-stakes, on-demand assessment

Future direction of work n n Run exams with full year groups Refine question authoring Evaluate other freetext question formats currently used within the medical school Investigate the possibility of distributing tests asynchronously throughout the curriculum