ITEM ANALYSIS Dr Umed V Patel Associate Professor

ITEM ANALYSIS Dr. Umed V Patel Associate Professor Dept. of Community Medicine, PDU Government Medical College, Rajkot 1

OBJECTIVES At the end of this session the learner should be able to: 1. Define item analysis and to understand the purpose of item analysis 2. Explain the procedure for calculation of Difficulty index, Discrimination index and Distracter effectiveness. 3. Review MCQs and make judgment on the basis of item analysis and suggest improvements for defective items. 4. List uses of item analysis 2

What do you like? a. b. c. d. Movies Cricket Statistics All of the above 3

What is common theme in all four photographs What is your analysis? 4

MCQs (Items) • Objective evaluation is becoming more important and popular in education • With greater usage of MCQs in various kind of examinations, Large no. of MCQs (Question bank) with different level of difficulty requires. • In addition, MCQs are used to assess class performance as a part of formative evaluation. Formative evaluation: It is to determine how much and how well students have learned, to serve as a feedback to students & teachers. 5

VALIDATION It is an important step in formulation of MCQs. Validation require at TWO stages. • Pre-validation: It is a process to which a constructed item is subjected prior to appearing in an examination. • Post validation: It is the process of analysis of the item after it has appeared in a test. (item analysis) 6

DEFINITION • Item analysis is the process of analyzing the performance of a MCQ after it has appeared in a question paper. • Statistical procedure is require to analyze a question (item) 7

Purpose of Item analysis: 1. Whether the item is of appropriate level of difficulty 2. Whether the item is capable of discriminating between the knowledgeable and less knowledgeable students. 3. To get the “functionality” of alternatives(options) to the corrected response 4. To create question bank with known difficulty and discrimination capacity. 8

Distracter Effectiveness Discrimination Index Difficulty Index Item Analysis Provides: …. 9

Item Analysis Provides: …. 1. Difficulty Index (P) 10

PROCEDURE The essential steps of item analysis are: 1. Score the whole test for all the students. 2. Rank the students in order of merit based on their test scores. 3. Take the top third as high achievers and bottom third as low achievers. 11

4. Prepare a table for each item as follows: Example. Item No 1 ` Option No. selecting s the option amongst high achievers (H) Key (correct response “C”) No. selecting the option amongst low achievers (L) Total response N(%) A 5 10 15 (15%) B 5 10 15 (15%) C 30 10 40 (40%) D 10 10 20 (20%) E Nil 2 2 (2%) Not responde d Nil 8 -- Total (N) 50 12 50 100

5. Calculate various indices 1. Difficulty index Indicated by the symbol ‘P’ and is calculated by the formula P = H + L x 100 N Where, H = No. of students answering the item correctly in the high achieving group. L = No. of students answering the item correctly in the low achieving group. N = Total number of students in two groups including nonresponders. 13 Thus, for the given example the value of ‘P’ = 30 + 10 x 100 =

Interpretation of Difficulty Index (P) P Value Interpretation 50 -60% Good 30 -70% Acceptable <30% & >70% Requires Modification (too difficult or too easy 0 10 20 30 40 50 60 70 80 90 100 14

2. Discrimination index indicated by the symbol ‘d’ and is calculated by the formula d=H-L x 2 N Where, H = No. of students answering the item correctly in the high achieving group. L = No. of students answering the item correctly in the low achieving group. N = Total number of students in two groups including nonresponders. Thus for the given example the value of ‘d’ = 30 - 10 x 2 = 0. 4 100 15 Simply,

Interpretation of Discrimination Index [d] d Value ≥ 0. 35 0. 25 to 0. 34 0. 20 to 0. 24 <0. 20 Interpretation Excellent discrimination Good Acceptable Requires Modification . _________. -1 0 0. 25 0. 35 1 A negative Discrimination Index almost always indicates a wrong item/key 16

3. Distracter effectiveness • Any of the distracters in the item, which has not attracted even 5% of the total responded item, is said to be a “non-functional” distracter. • In the given example, option number ‘E’ is nonfunctional having attracted only 2% of students. So, “alternative E” should be changed 17

Item Analysis – An Example 18

1. Award of a score to each student ROLL NO. MARKS 1 16 13 15 25 14 37 16 2 13 14 8 26 22 38 7 3 21 15 9 27 8 39 18 4 12 16 20 28 15 40 13 5 11 17 9 29 8 41 19 6 15 18 17 30 23 42 11 7 24 19 9 31 12 43 9 8 19 20 18 32 10 44 12 9 10 21 16 33 12 45 17 10 14 22 11 34 21 46 13 11 12 23 20 35 14 47 16 12 18 24 10 36 15 48 20 19

2. Ranking in order of merit Marks Roll No. 25 - 13 2, 40, 46 24 7 12 4, 11, 33, 44 23 30 11 5, 22, 42 22 26, 10 9, 24, 32 21 3, 34 9 15, 17, 19, 43 20 16, 23, 48 8 14, 27, 29 19 8, 41 7 38 18 12, 20, 39 6 17 18, 45 5 - 16 1, 21, 37, 47 15 6, 13, 28, 36 14 10, 25, 35 20

3. Divide the student in 3 groups - on the basis of performance Question -1 20 th June 2010 Total Students : 48 Higher Group 24, 23, 22, 21, 20, 20, 19, 18, 18, 17, 16 Middle Group 16, 16, 15, 15, 14, 14, 13, 13, 12, 12 Lower Group 12, 11, 11, 10, 10, 9, 9, 8, 8, 8, 7 Q. 1 RD A SC L E DI IDD UP M O GR Frequency Table Alternatives Responses of Higher Group Responses of Lower Group a 2 3 b 5 8 C 9 =H 3=L d - 1 Not responded - 1 Total 16 16 21

Calculation of Indices Q-1 H=9, L=3 N = 32 Difficulty Index (P) (expressed as %) P = H + L X 100 N So, P = 9 + 3 X 100 = 32 37. 5% ============================= Discrimination Index (d) (expressed As A Decimal Fraction) d=H-L X 2 N So, here, d = 9 – 3 X 2 = 32 0. 39 22

Distracter Effectiveness or functionality Effectiveness of the options Any of the distracters(alternatives) in the item which has not attracted even 5% of the total responses is said to be a nonfunctional distracter Alternatives Responses of Higher Group Lower Group Total responses a 2 3 5 (16. 1%) b 5 8 13 (41. 9%) C 9 =H 3=L 12(38. 7%) d - 1 1(3. 2%) Not responded - 1 --- Total 16 16 31 (100%) Alternative d have attracted <5% student. So, it is nonfunctional and require modification 23

Question Bank Card Subject Objective Tested Question: …. Alternatives (A) (B) (C) (D) Correct Answer : Stem Year P value Item Type & Action d value Item Type & Action 24

Question Bank Card Subject : Autonomic Nervous System-glaucoma Objective Tested : Able to name drug/S contraindicated in narrow angle glaucoma Question No. 1 Which of the following drug is contraindicated in narrow angle glaucoma? Alternatives a) Phenylephrine b) Timolol c) Homatropine d) Latanoprost Correct Answer : (C) Stem Year P value IV 2010 38. 7% Item Type & Action Acceptable d value 0. 39 Item Type & Action Excellent Retain 25

Uses of Item analysis 1. To determine if item function as expected 2. To improve item quality 3. To provides data/information for helping the students to improve their learning in class room teaching 4. Creation of an item bank with different level of difficulty 26

Thank You But now, its your turn to bat 27

ITEM ANALYSIS – GROUP TASK Three hundred students were administered an MCQ test containing 200 items. The frequency distribution table of the following items showing the number of students in the upper and lower group (100 each) who selected each alternative are prepared. Number underlined is response to correct answer. Calculate Difficulty Index, Discrimination Index and Distracter Effectiveness of the item allotted to you. Group Q TOP GROUP BOTTOM GROUP No A B C D NR Group 1 1 3 19 5 5 68 1 11 12 3 73 Group 2 2 51 23 23 2 1 25 14 42 11 8 Group 3 3 1 2 93 0 4 33 17 5 13 32 Group 4 4 14 60 14 10 2 24 35 29 6 6 28

ITEM ANALYSIS – GROUP TASK Calculate Difficulty Index, Discrimination Index and Distracter Effectiveness of the item allotted to you…. . P = H + L x 100 (30 -70% Acceptable, <30%/>70% require modification) N d = H - L x 2 (>0. 35 Excellent, <0. 20 require modification) N Group Q No TOP GROUP BOTTOM GROUP A B C D NR Group 1 1 3 19 5 5 68 1 11 12 3 73 Group 2 2 51 21 23 2 3 25 14 42 11 8 Group 3 3 1 2 93 0 4 33 17 5 13 32 Group 4 4 8 60 14 10 8 24 35 19 6 16 29

Thank You Again 30