BILC Standardization Initiatives and Conference Objectives Peggy Garza

BILC Standardization Initiatives and Conference Objectives Peggy Garza Associate BILC Secretary For Testing Programs

Outline • Standardization Efforts - Language Testing Seminars - Tailored Workshops - Toolbox for Language Testers - STANAG 6001 Testing Conferences • Conference Topics and Goals

BILC Testing Seminars • Language Testing Seminar (LTS) • Advanced Language Testing Seminar (ALTS) • Language Standards and Assessment Seminar (LSAS)

LANGUAGE TESTING SEMINAR (LTS)

Language Testing Seminar (LTS) • Developed and conducted by BILC testing experts • Background: A BILC initiative aimed at promoting a common interpretation of STANAG 6001 and consistency in language proficiency testing • Specific objective: To give NATO/Pf. P language teaching professionals the opportunity to practice developing test items based on STANAG 6001

Language Testing Seminar (LTS) • Held since Autumn 2000 at the Marshall Center, Garmisch, Germany • Two-week graduate-level seminar • Offered twice times a year (February and September) • More than 350 graduates from NATO/Pf. P/MD nations

ADVANCED LANGUAGE TESTING SEMINAR (ALTS)

Advanced Language Testing Seminar (ALTS) • Need for continuing professional development for test developers identified • Stakeholders’ meeting in September 2008 • Result: Advanced Language Testing Seminar (ALTS) • Validated in 2009

Advanced Language Testing Seminar (ALTS) • Target audience: LTS graduates who continue to work on STANAG 6001 testing - STANAG 6001 Edition 4 standardization sessions - Focus on Level 3 for practical exercises - Opportunity to exchange best practices • Modular approach to seminar

Advanced Language Testing Seminar (ALTS) • Pre-seminar assignments - Approximately 40 hours • Three-week seminar - Module 1: Receptive Skills: Listening and Reading (1 week) - Module 2: Productive Skills: Speaking and Writing (1 week) - Module 3: Test Analysis (3 days) - Module 4: Managerial Issues (2 days)

NEW LANGUAGE STANDARDS AND ASSESSMENT SEMINAR (LSAS)

Language Standards and Assessment Seminar (LSAS) 3 -11 July 2013 • For language teachers and program managers • Objectives – – STANAG 6001 familiarization Guidelines for aligning instruction to standards Test and measurement principles Classroom assessment and practical exercises

Testing Seminar Schedule in 2014 • Language Testing Seminar (LTS) ‒ 10 -21 February ‒ 15 -26 September • Advanced Language Testing Seminar (ALTS) ‒ 2 - 20 June • Language Standards and Assessment Seminar (LSAS) ‒ 23 Apr – 1 May ‒ 3 -11 December

STANAG 6001 Toolbox for Testers • Recommended by ALTS participants – Samples of listening and reading items at levels 1 -3 – Samples of speaking and writing tests at levels 1 -3, including plus levels – Frequently Asked Questions (FAQs) – Self-Assessment Outline for STANAG 6001 Testing Programs

BILC Website www. natobilc. org

Tailored Workshops for National Testing Programs

Tailored Workshops • Ukraine ‒ November 2011, 2012 • Moldova ‒ May 2012 • Russia ‒ June 2013

STANAG 6001 Testing Conferences • Zagreb, 2009 – Focus on standardization, moderation, and validation • Sarajevo, 2010 – Dates: 31 August – 2 September – Focus on testing speaking • Stockholm, 2011 – Dates: 6 -8 September – Focus on testing writing • Copenhagen, 2012 – Dates: 3 -6 September, 2012 – Focus on testing listening

2013 STANAG 6001 Testing Conference • Some of your questions: – Our stakeholders have questions. Is there advice for how best to communicate with the stakeholders? – Our stakeholders are asking us to test more and more people. What should we tell them? – We have developed items. What’s next? – How do we know if our tests are good? – We need to hire new testers. Is there a position description for testers? How can we recruit for testers? – How can we keep professionally current ?

2013 STANAG 6001 Testing Conference • Conference format – Presentations – Guest speakers from NATO and SHAPE – Panel discussions o Resolving challenges – Study Groups meetings and reports o Making a validity argument o Washback o Action plans o Renorming and standardizing testers o Piloting items and prompts

Specific Conference Goals • Review and clarify the entire test construction process • Identify action plans for test development and administration • Provide ideas for selecting the appropriate testing techniques • Discuss piloting and pretesting • Learn more about determining test validity • Identify practicality issues and discuss lessons learned • Discuss best practices in renorming and standardizing testers • Learn about innovations in testing in the BILC community – Computer-delivered testing – Proficiency and specialist testing – Testing at Level 4

Test Life Cycle Diagram

Communicating with the Stakeholders • Identify the stakeholders • Coordinate with the stakeholders during the needs analysis and planning stages – Stakeholders approve personnel and resources • Close the loop with the stakeholders during the Test Life Cycle – – Action plans Technical reports Updates Washback

Test Construction Stage 1 Needs Analysis Stage 2 Planning Stage 3 Design Stage 4 Development Test Specifications Revision Pre-testing / Analysis Standard Setting Evaluation/review Stage 5 Administration Stage 6 Monitoring Evaluation

Development Stage Part 1 Text selection Item writing Item review I (moderating) Piloting on a small scale Item review II (editing) • Retain • Revise • Reject Assembly of trial test More items needed?

Development Stage Part 2 Pre-testing/trialling on a larger scale Statistical Analyses • Retain Item revision • Revise • Reject Standard Setting Item banking Assembly of final version of test

2013 STANAG 6001 Testing Conference • Expectations – – Active participation and interaction Frank and open discussions Provide answers to the difficult questions Create a forum for future networking and cooperation • BILC appreciates your participation and we look forward to a very productive conference!