Frameworks Standards and Scales in the USG ECOLT
- Slides: 27
Frameworks, Standards, and Scales in the USG ECOLT 2009 November 6, 2009 ILR Testing Committee 1
Role of the ILR Testing Committee v Provide forum for discussion v Build network among government, academe and industry v Develop understanding of Skill Level Descriptions
Characteristics of the ILR Scale v Non-negotiable frame v Mastery principle v Proficiency not performance or achievement v Global rating v Apex is well-educated native speaker 3
Overview of Presentation v Use of the ILR Self Assessments v OPI Summits v First Listening Summit v ASTM Standard Practice 4
Frameworks, Standards and Scales Impact of Internet-based Speaking Self -Assessment Tool on Department of State New Hires
Setting Until September 2007 – one pencil and paper Foreign Service Examination/year resulting in: v Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service
Setting: applicant testing v Since September 2007 – 4 online Foreign Service examinations/year resulting in: v Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service plus v Approximately 1000 additional tests in Super Critical Need Languages: Arabic, Chinese, Hindi, Urdu, Dari, Farsi
Setting: expectations Expected threshold level for testing in: v World languages: Ø Speaking Proficiency 3 v All other languages: Ø Speaking Proficiency 2
Challenges v High testing numbers with strict deadlines v Low passing rates
Solution Internet-available Speaking Self-Assessment Tool on the ILR Homepage http: //www. govtilr. org/
Results v Before Speaking Self-Assessment Tool: Applicants tested: 85% Passing rate: 42% v After Speaking Self-Assessment Tool: Applicants tested: 63% Passing rate: 63%
Additional Results v More realistic expectations v Happier examinees v Test administration resource savings.
Frameworks, Standards and Scales Oral Proficiency Interview Summits Listening Summit
Impetus for Speaking and Listening Summits v Started in June 2008 as a result of ACTFL OPI work for DLI at Mid & Higher levels v Questions about comparability at the higher levels in Speaking v Success of Speaking Summits led naturally to Listening v Opened up to the wider ILR community and non-USG 11/30/2020 9: 22: 27 AM 14
Summit Overall Goals v Create common understandings v Achieve comparable scores v Supplement the ILR Skill Level Descriptions v Long-term goal to update the SLDs within the ILR Framework 11/30/2020 9: 22: 30 AM 15
OPI Summit Highlights v Highlighted similarities and differences in approaches to testing Speaking v Recognize that agencies’ tests are different, but all aim to measure speaking in the ILR context v Uncovered differences in interpretations of the SLDs v Discussion on keeping the Educated Native Speaker at the Apex of the Scale v Agreement to continue to tighten wording and standards 11/30/2020 9: 22: 31 AM 16
OPI Summit Outcomes v Work underway to write explanatory appendix of terms v TAEG-Sponsored study of OPI comparability has been designed and is waiting approval 17
Listening Summit Goals v Surface areas in the SLD that need our attention v Build a common understanding on the challenge of listening proficiency 11/30/2020 9: 22: 34 AM 18
Listening Summit Outcomes v Identified participative and nonparticipative listening as a major issue v Generated a list of terms that need to be further defined and/or clarified v Recognized need to bring in new research in the field 19
Next Steps v OPI Summit work is underway to draft explanatory notes for Speaking Skill Level Descriptions v Next Listening Summit in the winter 2010 20
Frameworks, Standards and Scales Development of the ASTM Standard Practice
What is ASTM? • ASTM: American Society for Testing and Materials Ø 30000+ industry standards Ø Develops but does not enforce standard Independent, private sector, not-for-profit v Mission is to provide a system for experts to directly participate in developing market relevant, high quality, international standards v Standards developed in an open process involving all interested parties (government & industry) v ASTM does not enforce standards v Standards may be referenced in contracts
The ASTM Standard v Many ILR users are competent test developers and can design their own test specifications v BUT Ø Some ILR users need a test but do not have the expertise to develop it or to determine what existing tests might suit their need Ø ILR users might differ in their interpretations of the ILR for testing; these differences might not be apparent in test specifications
The ASTM Standard v Having an overarching standard allows multiple test developers all referencing the ILR to have a common reference point v A formal standard is transparent and allows contractors and government agencies to know what is required v Standard is a basis for quality assurance v Standard guide vs. standard practice vs. standard specification
Challenges v The ASTM Standard Practice is NOT an interpretation of the ILR v As of now, there is no enforcement mechanism for the standard v The standard is modular, since different testing needs require different types of tests and test development processes— not lockstep standardization
Looking forward v The process of articulating the standard has brought about unprecedented dialogue among different agencies and contractors about what is expected and needed v The requirements for needs analysis and frameworks that are stated in the standard practice raise awareness among stakeholders that will lead to better-fitting tests
Conclusions v Broaden access to the ILR framework v Improve common understanding of the ILR scale v Update Skill level Descriptions 27
- Nursing theories
- Nursing informatics theories, models and frameworks
- Food security concepts and frameworks
- Beneficence
- Strategic management frameworks
- 3 example of paragraph
- Contemporary frameworks
- Parcc model content frameworks
- Java ecommerce framework
- List of theoretical frameworks
- Architecture frameworks
- Enterprise agile frameworks
- Why i hate frameworks
- Social studies frameworks
- Describe trust frameworks.
- Actor frameworks
- A level english language frameworks
- Software architecture frameworks
- Post positivism
- Regional construction frameworks
- Interpretive framework example
- Slidetodoc
- Net frameworks 4
- Local development frameworks
- Framework of knowledge
- Preemptive message strategy
- Factors determining service standards
- Invaginasi usg