PERFORMANCE CHARACTERISTIC OF COMPUTER BASED EXTENDED MATCHING ITEMS

PERFORMANCE CHARACTERISTIC OF COMPUTER BASED EXTENDED MATCHING ITEMS AND ONE BEST ANSWER: A PILOT STUDY Lunelle L Pienaar & Gregory Doyle Education Development Unit, Faculty of Health Sciences, University of Cape Town

o Multiple choice format remain one method of assessment that is most commonly used to test theoretical knowledge in medical education (Palmer EJ and Devitt PG. 2007, Mavis et al, 2001). o Increasingly computer based assessment is used more within the FHS UCT-student numbers, drive increase IT use

o R-type (OBA) o Traditionally used more often and is written with question in the form of a clinical case scenario with 4 or 5 option, questions are thought to test recall of knowledge. A-type (EMI) The questions are written with a theme and questions are constructed around the focus theme (Beullens et al, 2002, Palmer and Devitt, 2007).

o EMIs is considered a better format to test problem solving and integration of knowledge( Swanson et al, 2008, Swanson et al, 2006). o EMI has demonstrated better discrimination between students as well as more difficulty compared to OBA (Mc. Coubrie 2004).

o What is the performance of in house test administered to medical students during one of their clinical rotations in 5 th year. o What is the test item difficulty and discrimination of EMIs and OBA used in summative assessment. o Is there any differences between item difficulty and discrimination between the EMI and OBA.

o The questions were administered to all 5 th year students at the end of their clinical block. o We evaluated OBA items and extended matching items that were developed by the 5 th year course lecturer and evaluated as appropriate for testing by the course convener(not the researcher). o The item difficulty and discrimination for the EMIs and OBA was determined

o o o Results were from the end of block test with 5 groups of students. The same questions were administered to students all 5 groups of students. The 193 students answered a computer based test containing five EMI sets with 34 items and 16 (OBA)-Type MCQs. The OBA -had 4 options with each of the 5 EMIs sets had 9 options with 9 questions.

71. 0% 80. 0% 70. 0% 60. 0%50. 0% 31. 0% 50. 0% 40. 0% OBA 20. 5% 19. 0% 30. 0% EMI 8. 8% 20. 0% 10. 0% >75% Easy 50 -75 % Moderate 20 -49 % Difficult < 20 Very difficult 0. 0%

69% 70% 52% 60% 47% 50% 40% 30% OBA 25% EMI 20% 6% 10% 0% 0% < 0. 20 0. 2 -0. 45 0

Item difficulty Item Discrimination Mean Std Dev OBA (N=16) 0. 71 0. 22 0. 24 0. 11 EMI (N=34) 0. 79 0. 19 0. 23 0. 12

o The study did not show any statistical significance between the 2 types of MCQs – small sample of test items. o Possible reason for this may also be that writing OBA format were used more often in the past. o o o Both EMI and OBA test items wide range of item difficulties The results raises 2 issues when questions are reused to select the best performing items. o evaluation of test post administration is important to determine how to improve quality of the assessment items. o

o Beullens J, Van Damme B, Jaspaert H, Janssen PJ. 2002. Are extended-matching multiple-choice items appropriate for a final test in medical education? Medical Teacher 24 (4): 390 -5. o Mavis BE, Cole BL, Hoppe RB. 2001 A survey of student assessment in U. S. medical schools: the balance of breadth versus fidelity. Teaching and Learning in Medicine, 13 (2), 74 -79. o Mc. Courbrie P. 2004. improving the fairness of multiple- choice questions: a literature review. Medical Teacher, vol 26. 8 (709 -712) o Palmer EJ and Devitt PG. 2007. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ; 7: 49. o Swanson, D. B. , Holtzman, K. Z. , Allbee, K. 2008. Measurement characteristics of content-parallel single-best-answer and extended-matching questions in relation to number and source of options. Journal of the Association of American Colleges, 83(10 Suppl): S 21 -4 o Swanson, D. B. , Holtzman, K. Z. , Allbee, K. , Clauser, B. E. 2006. Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options. Journal of the Association of American Colleges. Abstract 81(10 Suppl): S 52 -5

http: //edusum. edublogs. org/category/thought s-on-assessment/