BILC PROFESSIONAL SEMINAR Stockholm October 14 17 2013
BILC PROFESSIONAL SEMINAR Stockholm , October 14 - 17, 2013 INNOVATIVE TEST DESIGNS AND FORMATS Testing in the time of crisis Lt. Col. F. Gratton Francesco Gratton 2013
Summary: • Past situation (up to Sept 2013) • New Course Of Actions adopted • A do-it-all software • Proposals Francesco Gratton 2013
Stanag Proficiency Test 1. 0 RECEPTIVE SKILLS (Listening & Reading) 1: Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5: Separate Sections (& levels) 6: # of correct answers multiplied for a coefficient 7: Potential use of “F” factor Francesco Gratton 2013
How STANAG levels were awarded (Stanag Proficiency Test 1. 0) RECEPTIVE SKILLS # of correct answers Multiplied by fixed coefficient (1, 66) Francesco Gratton 2013
Francesco Gratton 2013
Proficiency Test 1. 0 PRODUCTIVE SKILLS SPEAKING (holistically assessed ) 1: Functional language assessed in a global manner 2: Structured interview 3: Tailored to the candidate 4: Checks & probes 5: 1 to 2 role plays Francesco Gratton 2013
JFLT 1. 0 PRODUCTIVE SKILLS WRITING (holistically assessed ) Three tasks (one for each level) Francesco Gratton 2013
Summary: • Past situation • New COAs adopted • A do-it-all software • Proposals Francesco Gratton 2013
New COAs adopted • Specifications • Cut-off score • Joint Database Francesco Gratton 2013
Francesco Gratton 2013
• PURPOSE • LEVELS OF LINGUISTIC KNOWLEDGE • TEST CONTENT • TEST FORMAT • VALIDATION PROCEDURES • ADMINISTRATION PROCEDURES Francesco Gratton 2013
Stanag Proficiency Test 1. 0 TWAS HIGH & Reading) RECEPTIVE SKILLS (Listening 1: TIME!!! Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5: Separate Sections (& levels) 6: # of correct answers multiplied by a coefficient 7: Potential use of “F” factor Francesco Gratton 2013
Stanag Proficiency Test 2. 0 New Key-factors • Each section is a mini-test (L & R) • Plus levels • Percentages Francesco Gratton 2013
New COAs adopted • Specifications • Cut-off score • Joint Database Francesco Gratton 2013
JFLT 2. 0: RDS (Listening & Reading) Section 1 –Stanag level 1 (questions from 1 to 15) No. Correct answ. 1 2 3 4 5 6 7 0 Level awarded 8 9 10 11 12 13 14 15 0+ 1 Section 2 –Stanag level 2 (questions from 16 to 30) No. Correct answ. 1 2 3 4 5 6 7 1 Level awarded 8 9 1+ 10 11 12 13 14 15 2 Section 3 –Stanag level 3 (questions from 31 to 45) No. Correct answ. 1 2 3 4 5 6 7 2 Level awarded 8 9 2+ 10 11 12 13 14 15 3 Section 4 –Stanag level 4 (questions from 46 to 60) No. Correct answ. 1 Level awarded 2 3 4 3 5 6 7 8 3+ 9 10 11 12 13 14 15 4 Francesco Gratton 2013
EXAMPLE ü Level 1 (15 questions) ü Level 2 (15 questions) ü Level 3 (15 questions) Max score: 45 All candidates answer correctly to 30 questions With Old test they would all get the same score Francesco Gratton 2013
CORRECT LEVEL 1 LEVEL 2 LEVEL 3 BIANCHI 30 10 10 10 ROSSI 30 11 10 9 VERDI 30 13 12 5 GIALLI 30 15 8 7 ARANCIONI 30 15 6 9 CANDIDATO ANSWERS (15 Questions) FINAL LEVEL 3 2+ 2 1+ 1 Francesco Gratton 2013
New COAs adopted • Specifications • Cut-off score • Joint Database Francesco Gratton 2013
NEW LISTENING & READING ITEMS JDB (JOINT DATA BASE) JOINT EFFORT HOW NOT TO MAKE JOINT EFFORTS Francesco Gratton 2013
Test-writers involved • Accustomed to military environment • Language Testing Seminar • Qualified • Norming Sessions Francesco Gratton 2013
ARMY CARABINIERI E H T NAVY B D J W O FL AIR FORCE Francesco Gratton 2013
h t 4 up o Gr AUGUST 2013 60 NEW ITEMS. INTO JDB Phase 1 NOVEMBER 2013 80 NEW ITEMS. INTO JDB Phase 4 Phase 2 Phase 3 P VI RE-T SIO E S 50 N T TIN R G (- ITEM IALL 30 %) S IN# Phase 3 RE Phase 2 M F AP ODI INA PR FIC L OV AT AZ IO IO N NE RE PR VI ESIO TE 50 N T STIN R G (- ITEM IALL 30 %) S IN# Phase 4 T FO EST R 1 ERS ° R M EV EE ISI T ON RE PR VI E-T SIO E S 50 N T TIN R G (- ITEM IALL 30 %) S IN# T FO EST R 1 ERS ° R M EV EE ISI T ON TIMINGS Phase 3 Phase 1 • R • P EPE # RE RIM • IN 120 PAR EN VI TTI AZI TO O V O GR AD AZI NE UP AL ON PO TR I O 3 r up o Gr d Phase 2 • P R FIR EPA ST RA • SE I BAT TION NT TEM CH O O F F GR TO O S OU TH P ER d n 2 up o Gr Phase 1 • P R FIR EPA ST RA • SE I BAT TION NT TEM CH O O F F GR TO O S OU TH P ER t 1 s u p o Gr Phase 1 • P R FIR EPA ST RA • SE I BAT TIO NT TEM CH N O O F F GR TO O S OU TH P ER 2012 AUG SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL WG Gd. L 2013 FEBRUARY 2013 20 NEW ITEMS. INTO JDB MAJ 2013 40 NEW ITEMS. INTO JDB Francesco Gratton 2013
Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals Francesco Gratton 2013
WHAT’S THE DIFFERENCE ? Francesco Gratton 2013
PC-assessment-related terminology Term Definition Any systematic method of obtaining Assessment evidence (through questions) for a purpose. Stakes Quiz … measures for the purpose of providing Low feedback to the student. Survey … to determine needs required to fulfill a Low defined purpose. Test Exam … measures knowledge for the purpose of informing the student on their current level Medium … measures knowledge for the purpose of High documenting the current level of knowledge Francesco Gratton 2013
the software is used for: • Needs analysis: (surveys) • Placement test • Any training activity • Assessment: – First level survey – Post-course – Pre-certification – Certification Francesco Gratton 2013
Software’s system Schema Create questions And organize them in tests using a windows based PC Francesco Gratton 2013
Assessment …… … allows to choose: On Windows PC • Time limits • Feedback to test-taker • Styles (Template) Questions via Browser Assessment Definitions • Jumps • Question shuffling • Instructions to testtakers Francesco Gratton 2013
Assessments • … also created with su PC Windows authoring manager by selecting Qs previously created • Any question can be Questions via Browser Assessment Definitions chosen from the database Francesco Gratton 2013
Software’s system Schema Result reports, CIA, graphs, gimmicks, you name it …. . assessment published using any browser, secure browsers, or a PC/MAC Create questions And organize them in tests using a windows based PC Set security parameters, schedule assessment and link to other (Learning Management Systems Francesco Gratton 2013
Types of Questions: Multiple Choice Francesco Gratton 2013
Likert Scale (for questionnaires) Francesco Gratton 2013
Essay Question 1. Candidate can write free text in the space provided 2. Testers will evaluate later Francesco Gratton 2013
Francesco Gratton 2013
Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals Francesco Gratton 2013
Testing in the time of crisis • Wide project • Sharing experience & capabilities • Optimization of resources • No alternative to B. A. T. Francesco Gratton 2013
Francesco Gratton 2013
COMBINED DATABASE or … A multilateral-based CDB or … A Bilateral-based CDB Francesco Gratton 2013
Francesco Gratton 2013
1 SPECS 2 3 FLOW TIME SCHEDULE Francesco Gratton 2013
Francesco Gratton 2013
canutesting@sclingue. esercito. difesa. it Thank you francesco. gratton@gmail. com Francesco Gratton 2013
- Slides: 42