Republic of Bulgaria Ministry of Defense Piloting STANAG

Republic of Bulgaria Ministry of Defense Piloting STANAG 6001 CAT Reading and Listening 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 1

Content • Plan for STANAG 6001 CAT implementation; • STANAG 6001 CAT software; • Evaluation algorithms – computer-assisted and computer-adaptive; • Piloting STANAG 6001 CAT for reading; • Piloting STANAG 6001 CAT for listening; • CAT – pros and cons. 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 2

Plan for STANAG 6001 CAT Implementation • since exam session Autumn 2013 – STANAG 6001 CAT and pen-and-paper tests; • since exam session Spring 2015 only STANAG 6001 CAT. 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 3

STANAG 6001 CAT software • • • Administrative Module main menus : Lists Protocols Tasks Tests Evaluation 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 4

6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 5

6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 6

6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 7

STANAG 6001 CAT software (cont. ) Candidates’ module: • Reading Test • Listening Test • Writing Test 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 8

Piloting STANAG 6001 CAT reading 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 9

Piloting STANAG 6001 CAT listening 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 10

Evaluation algorithm computer-assisted test • follows the current Specifications for STANAG 6001 exam for the Bulgarian Armed Forces; • starts with items for L 1, then L 2, then L 3 ; • level is given for min. 70% correct answers; • 50%-69% correct answers for the next level → (+) level. 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 11

Evaluation algorithm computer-adaptive test Each candidate is given 7 items for L 2 and depending on the correctness of the answers, the test continues moving further to L 3 or L 1 and again to L 2 if appropriate. The pass rate is 70% per level (50 -69% for + levels). 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 12

STANAG 6001 CAT Piloting READING and LISTENING Methods of validation: - statistical analyses of the performance of the participants comparing their performance at a recent pen-and-paper test and at the CAT; - retrospective written feedback by the participants in the piloting. 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 13

Piloting STANAG 6001 CAT Reading Results 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 14

STANAG 6001 CAT Piloting Reading Results 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 15

Piloting STANAG 6001 CAT Listening Results 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 16

STANAG 6001 CAT PILOTING LISTENING RESULTS 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 17

Participants’ Feedback Nearly 90% of the participants in the piloting approve the implementation of STANAG 6001 CAT for Reading and Listening. 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 18

CAT – pros and cons Pros: -candidates can manage their time and this makes them calm and confident; -decreased candidates’ frustration -immediate feedback; -highly objective feedback; -less items are compromised; -testing-time is saved; -new items can be piloted and the statistical calculations can be made automatically. 6 -9 May 2013 Tbilisi, Georgia Cons: -due to test method and conditions (e. g. computer screen clarity, motor skills needed to respond to test items, and ability of test administrators to troubleshoot); -due to test-taker attributes (e. g. computer familiarity, ability to concentrate in front of the computer, danger of bad time-management). Emilija Nesheva 19

NB! Lessons Learnt! 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 20

Questions? 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 21

May 6 th – St. George’s Day Valor Day – Day of Bulgarian Armed Forces 6 -9 May 2013 Tbilisi, Georgia Emilija Nesheva 22