Frameworks Standards and Scales in the USG ECOLT

  • Slides: 27
Download presentation
Frameworks, Standards, and Scales in the USG ECOLT 2009 November 6, 2009 ILR Testing

Frameworks, Standards, and Scales in the USG ECOLT 2009 November 6, 2009 ILR Testing Committee 1

Role of the ILR Testing Committee v Provide forum for discussion v Build network

Role of the ILR Testing Committee v Provide forum for discussion v Build network among government, academe and industry v Develop understanding of Skill Level Descriptions

Characteristics of the ILR Scale v Non-negotiable frame v Mastery principle v Proficiency not

Characteristics of the ILR Scale v Non-negotiable frame v Mastery principle v Proficiency not performance or achievement v Global rating v Apex is well-educated native speaker 3

Overview of Presentation v Use of the ILR Self Assessments v OPI Summits v

Overview of Presentation v Use of the ILR Self Assessments v OPI Summits v First Listening Summit v ASTM Standard Practice 4

Frameworks, Standards and Scales Impact of Internet-based Speaking Self -Assessment Tool on Department of

Frameworks, Standards and Scales Impact of Internet-based Speaking Self -Assessment Tool on Department of State New Hires

Setting Until September 2007 – one pencil and paper Foreign Service Examination/year resulting in:

Setting Until September 2007 – one pencil and paper Foreign Service Examination/year resulting in: v Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service

Setting: applicant testing v Since September 2007 – 4 online Foreign Service examinations/year resulting

Setting: applicant testing v Since September 2007 – 4 online Foreign Service examinations/year resulting in: v Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service plus v Approximately 1000 additional tests in Super Critical Need Languages: Arabic, Chinese, Hindi, Urdu, Dari, Farsi

Setting: expectations Expected threshold level for testing in: v World languages: Ø Speaking Proficiency

Setting: expectations Expected threshold level for testing in: v World languages: Ø Speaking Proficiency 3 v All other languages: Ø Speaking Proficiency 2

Challenges v High testing numbers with strict deadlines v Low passing rates

Challenges v High testing numbers with strict deadlines v Low passing rates

Solution Internet-available Speaking Self-Assessment Tool on the ILR Homepage http: //www. govtilr. org/

Solution Internet-available Speaking Self-Assessment Tool on the ILR Homepage http: //www. govtilr. org/

Results v Before Speaking Self-Assessment Tool: Applicants tested: 85% Passing rate: 42% v After

Results v Before Speaking Self-Assessment Tool: Applicants tested: 85% Passing rate: 42% v After Speaking Self-Assessment Tool: Applicants tested: 63% Passing rate: 63%

Additional Results v More realistic expectations v Happier examinees v Test administration resource savings.

Additional Results v More realistic expectations v Happier examinees v Test administration resource savings.

Frameworks, Standards and Scales Oral Proficiency Interview Summits Listening Summit

Frameworks, Standards and Scales Oral Proficiency Interview Summits Listening Summit

Impetus for Speaking and Listening Summits v Started in June 2008 as a result

Impetus for Speaking and Listening Summits v Started in June 2008 as a result of ACTFL OPI work for DLI at Mid & Higher levels v Questions about comparability at the higher levels in Speaking v Success of Speaking Summits led naturally to Listening v Opened up to the wider ILR community and non-USG 11/30/2020 9: 22: 27 AM 14

Summit Overall Goals v Create common understandings v Achieve comparable scores v Supplement the

Summit Overall Goals v Create common understandings v Achieve comparable scores v Supplement the ILR Skill Level Descriptions v Long-term goal to update the SLDs within the ILR Framework 11/30/2020 9: 22: 30 AM 15

OPI Summit Highlights v Highlighted similarities and differences in approaches to testing Speaking v

OPI Summit Highlights v Highlighted similarities and differences in approaches to testing Speaking v Recognize that agencies’ tests are different, but all aim to measure speaking in the ILR context v Uncovered differences in interpretations of the SLDs v Discussion on keeping the Educated Native Speaker at the Apex of the Scale v Agreement to continue to tighten wording and standards 11/30/2020 9: 22: 31 AM 16

OPI Summit Outcomes v Work underway to write explanatory appendix of terms v TAEG-Sponsored

OPI Summit Outcomes v Work underway to write explanatory appendix of terms v TAEG-Sponsored study of OPI comparability has been designed and is waiting approval 17

Listening Summit Goals v Surface areas in the SLD that need our attention v

Listening Summit Goals v Surface areas in the SLD that need our attention v Build a common understanding on the challenge of listening proficiency 11/30/2020 9: 22: 34 AM 18

Listening Summit Outcomes v Identified participative and nonparticipative listening as a major issue v

Listening Summit Outcomes v Identified participative and nonparticipative listening as a major issue v Generated a list of terms that need to be further defined and/or clarified v Recognized need to bring in new research in the field 19

Next Steps v OPI Summit work is underway to draft explanatory notes for Speaking

Next Steps v OPI Summit work is underway to draft explanatory notes for Speaking Skill Level Descriptions v Next Listening Summit in the winter 2010 20

Frameworks, Standards and Scales Development of the ASTM Standard Practice

Frameworks, Standards and Scales Development of the ASTM Standard Practice

What is ASTM? • ASTM: American Society for Testing and Materials Ø 30000+ industry

What is ASTM? • ASTM: American Society for Testing and Materials Ø 30000+ industry standards Ø Develops but does not enforce standard Independent, private sector, not-for-profit v Mission is to provide a system for experts to directly participate in developing market relevant, high quality, international standards v Standards developed in an open process involving all interested parties (government & industry) v ASTM does not enforce standards v Standards may be referenced in contracts

The ASTM Standard v Many ILR users are competent test developers and can design

The ASTM Standard v Many ILR users are competent test developers and can design their own test specifications v BUT Ø Some ILR users need a test but do not have the expertise to develop it or to determine what existing tests might suit their need Ø ILR users might differ in their interpretations of the ILR for testing; these differences might not be apparent in test specifications

The ASTM Standard v Having an overarching standard allows multiple test developers all referencing

The ASTM Standard v Having an overarching standard allows multiple test developers all referencing the ILR to have a common reference point v A formal standard is transparent and allows contractors and government agencies to know what is required v Standard is a basis for quality assurance v Standard guide vs. standard practice vs. standard specification

Challenges v The ASTM Standard Practice is NOT an interpretation of the ILR v

Challenges v The ASTM Standard Practice is NOT an interpretation of the ILR v As of now, there is no enforcement mechanism for the standard v The standard is modular, since different testing needs require different types of tests and test development processes— not lockstep standardization

Looking forward v The process of articulating the standard has brought about unprecedented dialogue

Looking forward v The process of articulating the standard has brought about unprecedented dialogue among different agencies and contractors about what is expected and needed v The requirements for needs analysis and frameworks that are stated in the standard practice raise awareness among stakeholders that will lead to better-fitting tests

Conclusions v Broaden access to the ILR framework v Improve common understanding of the

Conclusions v Broaden access to the ILR framework v Improve common understanding of the ILR scale v Update Skill level Descriptions 27