The SHAPE Update BILC Conference Istanbul 2010 Michael

  • Slides: 27
Download presentation
The SHAPE Update BILC Conference – Istanbul 2010 Michael Adubato SHAPE Language Testing Centre

The SHAPE Update BILC Conference – Istanbul 2010 Michael Adubato SHAPE Language Testing Centre Supreme Headquarters Allied Powers Europe 1

Always good to start off with a little humor! BILC Conference –May 2010

Always good to start off with a little humor! BILC Conference –May 2010

What’s been happening in beautiful Belgium… Belgian Prime Minister Yves Leterme Belgian Prime Minister

What’s been happening in beautiful Belgium… Belgian Prime Minister Yves Leterme Belgian Prime Minister resigns again Thu, 22 Apr 2010 14: 43: 58 GMT BILC Conference – May 2010

What’s been happening in beautiful Belgium… NATO has approved the new Peacetime Establishment (PE)!

What’s been happening in beautiful Belgium… NATO has approved the new Peacetime Establishment (PE)! And what does that mean? It means that they can start on the NEXT PE reorganization! BILC Conference – May 2010

What’s been happening in beautiful Belgium… What does this mean to us as language

What’s been happening in beautiful Belgium… What does this mean to us as language teachers, testers, etc? -It means that a bunch of military officers who have little or no knowledge about STANAG 6001 or what an individual is capable of doing at each level made command decisions about what SLPs are required for specific posts. -End Result: we have what appears to be a LOT more posts requiring 4’s in their profile! Again, what does this mean to us as language teachers, testers, etc? THE NEW YORK POST SAID IT BEST… BILC Conference – May 2010

BILC Conference – May 2010

BILC Conference – May 2010

Level 4 English language requirements in ACO (Allied Command Operations) consists of SHAPE JFC

Level 4 English language requirements in ACO (Allied Command Operations) consists of SHAPE JFC Brunssum and JFC Lisbon , JFC* Naples , as well as their subordinate commands. TOTAL NUMBER OF POSTS REQUIRING 4 s: 726 *Joint Force Command BILC Conference – May 2010 ,

Of the 726 posts, 65 of them need SLP 4444 SOME SEEM TO BE

Of the 726 posts, 65 of them need SLP 4444 SOME SEEM TO BE JUSTIFIED HOWEVER!! 22 posts at JFC Naples are for US military personnel (fine but next to impossible for most other nations to fill them if the USA does not!) (and I’m willing to bet that if those 22 were tested that all would NOT achieve SLP 4444 – half are NCO grades!!) Some interesting ones… -FRENCH OR 6 Staff Assistant in Travel Services – 4444 -USA OR 7 Supply Supervisor – 4444 -GERMAN OR 5 Data Base Manager – 4444 -GREEK OR 7 Admin Assistant – 4444 -FRENCH OR 6 Admin Assistant - 4444 BILC Conference – May 2010

The Solution? ? STUDY GROUP #4!!! In a nutshell: NATO tasked ACT who tasked

The Solution? ? STUDY GROUP #4!!! In a nutshell: NATO tasked ACT who tasked BILC who tasked STUDY GROUP #4 to come up with possible ways on how to proceed in sorting out this minor irritation. “Language Needs Analysis of Standard Language Proficiency levels in NATO Job Descriptions” BILC Conference – May 2010

Testing at SHAPE: Jun - Dec 2009 - Entry Tests Pass: 16 Fail: 12

Testing at SHAPE: Jun - Dec 2009 - Entry Tests Pass: 16 Fail: 12 Total: 28 42% FAILED! Total 38 78% FAILED! - Re-Tests - Pass: 8 Fail: 30 BILC Conference – May 2010

Testing at SHAPE: Jan-Apr 2010 - Entry Tests Pass: 8 Fail: 9 Total: 17

Testing at SHAPE: Jan-Apr 2010 - Entry Tests Pass: 8 Fail: 9 Total: 17 53% FAILED! Total 35 82% FAILED! - Re-Tests - Pass: 6 Fail: 29 BILC Conference – May 2010

Testing at SHAPE: Jan-Apr 2010 - Re-test Failures deficient skills: Listening: Speaking: Reading: Writing:

Testing at SHAPE: Jan-Apr 2010 - Re-test Failures deficient skills: Listening: Speaking: Reading: Writing: 3 14 3 16 For the 29 who failed! 14 candidates failed 1 skill, 10 failed 2 skills, 1 failed 3 skills and 5 failed 4 skills! BILC Conference – May 2010

Arriving at SHAPE with certificates -Total newcomers to SHAPE Jan – Apr 2010: 20

Arriving at SHAPE with certificates -Total newcomers to SHAPE Jan – Apr 2010: 20 -Arrived with certificates: 8 -Tested and passed (those w/o certif) : 2 -Tested and failed (those w/o certif): 3 -Pending 7 BILC Conference – May 2010

Correlation Study or Does TOEIC = STANAG 6001 French Navy HQs PARIS BILC Conference

Correlation Study or Does TOEIC = STANAG 6001 French Navy HQs PARIS BILC Conference –May 2010

Correlation Study in Paris Purpose – to see if there was any correlation between

Correlation Study in Paris Purpose – to see if there was any correlation between TOEIC (Test of English for International Communication) and STANAG 6001 The panel consisted of 15 individuals who were meant to be language testing/training experts There were FIVE of your distinguished BILC colleagues, a. k. a. “the usual suspects” who participated BILC Conference –May 2010

Mary. Jo Di Biase from BILC Conference – May 2010

Mary. Jo Di Biase from BILC Conference – May 2010

Corina Ispas from BILC Conference – May 2010

Corina Ispas from BILC Conference – May 2010

Karen Sacre, formally of DLI, San Antonio, USA BILC Conference – May 2010

Karen Sacre, formally of DLI, San Antonio, USA BILC Conference – May 2010

Ulla Gudnasson from (Homer loves Denmark!) BILC Conference – May 2010

Ulla Gudnasson from (Homer loves Denmark!) BILC Conference – May 2010

Yours truly from BILC Conference – May 2010

Yours truly from BILC Conference – May 2010

And some of our distinguished French colleagues from the different Language institutions in France

And some of our distinguished French colleagues from the different Language institutions in France BILC Conference – May 2010

Correlati 0 n Study Panel 1 – USA 1 – Italy 1 – Denmark

Correlati 0 n Study Panel 1 – USA 1 – Italy 1 – Denmark 1 – Romania 1 – ACO/SHAPE 3 – Poland 7 – France Flaw in the study – not everyone was familiar with STANAG 6001! BILC Conference – May 2010

Correlati 0 n Study Report -The panel seemed to be chose at random -Some

Correlati 0 n Study Report -The panel seemed to be chose at random -Some panel members had little or no experience with testing -One panel member had nothing to do with testing/training but was an administrator and POC for the study -TOEIC is a norm-reference d test* -TOEIC was developed in Japan (very different culture) for the purpose of testing Japanese & Korean in the business world *A norm-referenced test is a type of test, assessment, or evaluation which yields an estimate of the position of the tested individual in a predefined population, with respect to the trait being measured. This estimate is derived from the analysis of test scores and possibly other relevant data from a sample drawn from the Population. The term normative assessment refers to the process of comparing one test-taker to his or her peers. BILC Conference – May 2010

Correlati 0 n Study Report -The L &R parts focus on ONLY discreet points

Correlati 0 n Study Report -The L &R parts focus on ONLY discreet points (grammar & vocabulary) -Reading an English text was considered part of “speaking” test - Responses to computer-generated questions could be no longer than 30 seconds in duration, which does not show a ratable sample at any level (except perhaps L 1) -tasks are not able to reliably or accurately measure the language requirements IAW STANAG 6001 BILC Conference – May 2010

Recommendation -IGNORE any reports from ETS or any organization claiming a correlation between TOEIC

Recommendation -IGNORE any reports from ETS or any organization claiming a correlation between TOEIC & STANAG 6001 -IGNORE any STANAG certificates that were granted by nations (or anyone) based on taking the TOEIC BILC Conference – May 2010

The end… BILC Conference - May 2010

The end… BILC Conference - May 2010

? Questions? 27

? Questions? 27