Assessment Prof Tahmina Begum FCPS MD M M

Assessment Prof. Tahmina Begum FCPS, MD, M M Ed (UK)

Objectives Functions of assessment Domains of education Methods of assessment Criteria of a good assessment Advantages & limitations of written methods Construction of SAQ, MCQ Construction of OSCE/OSPE Mention the demerits of traditional oral examination Ø State the ways and means how the oral examination can be structured Ø Test construction Ø Ø Ø Ø

Assessment To change curricula or instructional methods without changing examination will achieve nothing G E Miller

Assessment ~ Learning -Gibbs

Why do we assess our students? • To ensure a minimum level of competence • To rank students competitively • To provide feedback to improve students learning (formative) • To evaluate teachers/ course • To certify

Types of Assessment Formative Summative

Types • Formative : attempts to answer the questions ‘How well is the student learning’ ? - Used to monitor progress - Detect learning error & provide feedback to student & tutor

Types (cont’) • Summative- (end of course) attempts to answer the question ‘ What has student already learnt’ ? - To ascertain students improvement - To rank student

Methods of assessment Methods Ø Written Ø Clinical/ Practical Ø Oral Instruments Essay, MCQ, MEQ, SAQ, Problem centered/ Case study exam, PMPs Long case, short case, OSCE/ OSPE Traditional viva/ SOE

DOPS, 360 degree evaluation, Performance Log, Portfolio Does OSCE, Long & Shows how Knows The assessment of clinical skills/ competence/ Performance (Miller GE) short case Standardized patient Clinical scenario based MCQ, Oral, Essay MCQ, Essay, Oral

Written method Used to test • knowledge • Application of knowledge

Cont’ • Response format a) Constructed/ open ended- candidate has to construct answer (fill in or multipage) e. g. Essay, SEQ, MEQ, Short answer, fill ins b) Selected/ closed ended, answers are provided in the form of options. E. g. MCQ, PMPs • Stimulus format- Problem centred/ case study exam

Cont’ • In the traditional open-ended essay-based assessment, the student is given a topic and instructed to write an extended descriptive narration • The expected reply may run into a few paragraphs (short note) or several pages (long essay)

Essay Question Advantage • Ability to write • Power of logic • Original thought • Creativity • Limitations • Low reliability • Subjective marking • Limited coverage

Example- Traditional: Write an essay on thyrotoxicosis. Total marks 25 OR A 46 year old lady presented with loss of weight, palpitation and weakness for 1 year. How will you proceed to diagnose the case? 25

Example: Structure of SEQ A 46 -year old female comes to you with a history of generalized weakness, palpitation, and excessive sweating for last 1 year. In spite of good appetite she has lost 10 kg of weight during this period. c 1. What other information will you take to obtain o for history? m 2. What additional physical signs will you look for p in this patient? o 3. What is the most likely diagnosis? n e 4. Mention 4 investigations you will advise to n support your diagnosis? t 5. Outline the steps in the management of this patient. Scenario 06 06 04 04 06

Short Answer Question (SAQ)

Types • Completion type: ü Incomplete statement ü Candidate supply word, term, symbol etc. ü Example 1. Hemophilia is due to def. of factor …………… 2. Define pulse ………………………. . 3. Draw/ label the intrinsic pathway of blood clotting 4. Name the Vit. K dependent clotting factors

• Open type: The statement should be worded clearly either in the form of direct question or following a short scenario 1. a) Enumerate 5 important causes of significant jaundice within 24 hours of life 2. 5 b) Outline the investigation plan with expected findings to reach a diagnosis 2. 5

Direct question Q 1. a) Name 4 Gm negative bacilli 2. 0 b) How do you differentiate endotoxin from exotoxin 3. 0 Q 2. a) What is DIC? 1. 0 b) Explain in short the mechanism of DIC 4. 0

Q 1. a) Classify diuretics 2. 0 b) Which diuretic will you use in a patient with heart failure and why 3. 0 Q 2. a) For bone age determination, X- ray of left wrist joint is advised. Why? Justify your answer 2. 5 b) How many bones will you expect in a 4 year old boy? 2. 5

• A 14 year old boy has been suffering from high grade fever for 1 month. On examination the boy was found severely pale with purpuric spots all over the body but there was no hepato-splenomegaly. 1. What is your clinical diagnosis? 2 2. Justify your answer 3

Reasoning • A 50 year old smoker has been suffering from high blood pressure for 1 year. He is diabetic for last 5 years. a) What antihypertensive drug will you use to control BP? Justify your answer. 3. 0 b) What will happen if BP is not controlled? 2. 0

Construction of Multiple Choice Questions (MCQ)

q. Multiple choice Questions (MCQs) are fixed response Objective assessment q. Widely used in under & post-graduate exam

Why we use MCQs? • • Purely objective in scoring & testing Broad coverage of content/ topics Efficient use of time High reliability Test large no. of candidate Ease of analysis Ease of marking - manually/ computer Banking

Limitations • • Allow for guessing Difficult & time consuming to construct Extended response Creative thinking can not be Writing ability assessed Longer reading time required Success depends on distracters

Types of MCQ • • Simple true false Multiple true-false type Single best answer Matching type Multiple true-false completion type Assertion and reason Extended Matching

Multiple True-false type • Candidate choose more than one response from a list of possible answers/ options • Options are related to single topic

Example Microcytic hypochromic anaemia is found in: A. Acute haemorrage B. Folate deficiency C. Iron deficiency Key- C, D, E D. Sideroblastic anaemia E. Thalassaemia

Following organisms are considered atypical a) b) c) d) e) Chlamydia Diphtheria Klebsiella Mycobacteriam Mycoplasma Key- a, e

Components of MCQ (Item) • Stem- present a problem/ a clinical scenario or a incomplete statement • Lead in- short question that the candidate is asked to answer • Option- 5 options should all be possible answer to the question • Correct answer is KEY, others are distracters

PLAN the question Recall Understanding • The learning level Application Clinical judgment problem solving

Options (key & distracters) • Similar in terms of grammar, length & complexity • Homogenous or fall into the same category e. g. signs, diagnosis, risk factors, tests, treatment etc. • The best No. is- 3 to 5 • Arrange the list of options systematically (chronological, alphabetical, numerical)

Other points need to consider Many DO’s & DO NOT’s

DO’s • • Clear and concise Reading time minimum Specify unit Use only plausible and logical distracters • Minimum negatively phrased statements in stem, options • Spell out acronyms, abbreviations

DO’s • • • Accurate in grammar & sentence Continuous with the stem Use familiar words Within the syllabus Important & useful aspect Link options to each other (e. g. all diagnosis, tests, treatment)

DO NOT’s/ Avoid • Too open /ambiguous term e. g. can / possible/may/sometimes/common/ often/seldom- likely to be true • Too absolute term e. g Never / always- tend to be false • Should not contain clue to answer • Avoid uncommon terminology & abbreviation • Avoid double negative

Avoid • • Implausible item More than one concept Incorrect information Too long statement Opposite statement in one branch Avoid clues to correct answer All or none of the above

Avoid Do not use the question-”which of the following statements are correct? ” The question is unfocused

SBA: Factual Recall of Knowledge The candidate repeats previously learned material by recalling facts, terms, and basic concepts Example: A branch of which cranial nerve supplies the vocal cords ? A. Optic B. Trochlear C. Abducens D. Vestibulocochlear E. Vagus

Scenario: A 5 year old girl has been brought to OPD for short stature Lead in: Select the hormones that are likely to be deficient here Options: a) Glucocorticoid b) Growth hormone c) Insulin d) Oestrogen e) Thyroxin Key : b, d, e

Scenario: A 8 year old boy developed respiratory distress who had puffiness of the face, scanty high colored urine and hypertension Lead in: Select the diuretic that should be administered for initial management in this case Options: a) Ameloride b) Frusemide c) Mannitol d) Spiranolactone e) Thiazide Key: b

SBA: Pathology Item A 72 -year-old man comes to the physician complaining of the sudden onset of knee pain that began a week ago. He describes the pain as sharp and severe and located in the joint. The knee is swollen, warm to the touch, and the skin over it is red. Joint fluid is aspirated, examined using polarized light microscopy, and shows the following: What is the most likely diagnosis for the patient’s knee pain? A. Gout B. Osteoarthritis C. Pseudogout D. Rheumatoid arthritis E. Traumatic effusion

The jugular foramen a) b) c) d) e) The accessory nerve passes Transmits inf. Petrosal sinus Is a foramen in occipital bone Is about 0. 5 cm in diameter Lies lateral to hypoglossal nerve × a –not gramatically continuous √ a) Gives passage to accessory nerve

Avoid using ambiguous/ imprecise/ absolute term Q 1. Microscopic exam of an area of inflammation: a) May reveal eosinophilic accumulation b) Sometimes shows lymphoid follicles Q. The possible side effects of digoxin are • Arrythmia • Nausea • Headache • -----× ambiguous / too open term ‘may’ / ‘sometimes’ /’possible’ have different meaning for different person

Cont’ a) Haemophillia never occur in female b) Hepatomegaly is always found in PEM × Avoid absolute term ‘‘never’/ always’ the answer is usually false a) Malarial parasite often found in blood film during febrile phase × ambiguous term ‘often’

The following are not found in thyrotoxicosis a) b) c) d) e) Weight gain Heat intolerance Reduced appetite Sinus tachycardia Goitre with bruit × Avoid negative in stem & if possible in item

O Objective S Structured C Clinical E Examination

OSCE / OSPE Ø Clinical competencies of medical students are assessed uniformly & objectively

The various competencies is assessed in different stations through which the candidates rotate until they complete a cycle

3 4 2 5 1 10 6 9 8 7

Organization…. • The basic structure of OSCE/ OSPE is a circuit of assessment stations • 10 - 20 stations are organized, students rotate on them • < 10 stations are inadequate to test all competencies • Student spends 5 minutes at each station

Organization …. 1. Procedure stations- students are asked to perform a procedure/ task (history taking, system exam, com skill) - requires an observer 2. Question stations- practically oriented questions are asked (data interpretation, image, picture, X-ray, write a prescription). - answers to be written on an answer sheet - scoring is done using a standard answer & marking scheme

Principle • A keen & silent examiner observes student • Observer scores the performance on a checklist (Heart of OSCE/ OSPE) which is prepared in advance by the consensus of examiners • Checklist is prepared by breaking the skill into vital components

Principle …. • The examiner stays at the station while the student move from station to station • All students are assessed on the same set of questions to make the assessment uniform & to ensure standardization

Some points to remember before construction of an OSPE Station

Domains Educational objectives are allocated to 3 domains Knowledge Attitude Skills

Levels of cognitive domain Evaluation Synthesis Analysis Application Comprehension Recall

Selection of Content Important topics should be considered first rather than a trivial one. v Must know v Useful to know v Nice to know

Nice to know Useful to know Must know

Points to be remembered q Language should be Clear, Simple & Easily Understandable q Instruction must be specific q Time & Task should be Consistent q Checklist should be prepared by breaking the expected skill into component parts q Proper Weight should be given to the important points in the checklist

Oral examination

Oral exam • An examination consisting of a dialogue with the examiner, who asks questions to which the candidate reply. • It assesses students’ ability to think and to express clearly his knowledge • It helps to assess other competencies such as communication skills and professional attitude • It takes the form of a series of interrelated questions

Cont’ • Oral examination is aimed at assessing mainly the higher degree of cognitive and problem solving skills of the candidates

Traditional oral exam (TOE) • TOE is a subjective test • Fails to assess properly the cognitive and problem solving skills • The atmosphere of the examination is often threatening • Independent marking by a pair of examiners is rarely done

Demerits of traditional oral exam o It lacks standardization, objectivity and reliability of results • Possibility of abuse of personal contact • There may not be enough adequately trained examiners • Expensive in terms of professional time • Language barrier may be a factor for poor performance • Test questions and the student responses go unrecorded

How can the oral examination be improved? • By organizing the oral examination in a structured manner • By structuring the oral exam: - elicit higher degree of cognitive, problem solving, interpretative and decision making skills - ensure higher degree of validity, reliability and objectivity

How can the oral examination be structured? The various elements of the oral examination may be organized:

The examiner • Careful selection of examiner and paring a new examiner with a more experienced person. • Briefing sessions for the examiners and providing the examiners with written instructions regarding the content area to be covered and the nature of competence to be measured.

The atmosphere § Creating a non threatening environment for candidate with equal duration of time for each candidate. § Starting with easy topic, then proceeding to more difficult problems. § Shifting of the topic in the situation, if someone fails to answer in one area

The questions • The questions are constructed by a group of faculty with inputs from all those who have participated in the teaching process. • A number of questions are prepared from each topic covering the content area of varying difficulty among the learning objectives.

Cont’ • The questions are then framed, few to assess the recall of facts but most to assess the problem solving abilities and higher cognitive domains. • Each question is typed on a card, divide the cards into groups and put in boxes of defined domain • Most correct answers for each question are decided in advance • Marks allocated for each parts of a question and weight for different level of difficulty is also to be decided.

The examination process § The candidate randomly selects one card from each box § Answers or clues are not provided there § The examiner reads the question, repeat if necessary or the candidate reads the question if allowed § After answering the questions the examiner place a tick in an appropriate box on a prepared rating scale § There is a scoring related to the boxes

Cont’ § If candidate fail to answer the first question, he will § § get no further chance in that area The examiners should not indicate whether the student answers correctly or not Each examiner should have equal time, each marking individually and scores are averaged afterwards Each question may be marked individually and summing up at the end Questions, answers and scores are noted concurrently by the examiners