Benchmark Advisory Test BAT Update BILC Conference Athens

Benchmark Advisory Test (BAT) Update BILC Conference Athens, Greece Dr. Ray Clifford and Dr. Martha Herzog 22 -26 June 2008

This Update Has Four Parts • Why we began the BAT project. • The role of proficiency standards. • Why the BAT follows a construct-based, evidence-centered test design. • When will BAT be available.

Why we began the BAT project. • 2006 A survey was conducted on the desirability of a BILC-sponsored, “benchmark” test with advisory ratings.

Participation in the Survey • 16 countries responded to the survey: Austria Bulgaria Canada Denmark Estonia Finland Germany Hungary Italy Latvia Lithuania Poland Romania Spain Sweden Turkey

Survey Results 1. Would your country use a Benchmark Test if one were available? Definitely yes: Probably yes: Perhaps: Most likely not: Definitely not: 8 5 2 0 1

Survey Results 2. Does your country use “plus levels” when assigning STANAG ratings? Definitely yes: Probably yes: Perhaps: Most likely not: Definitely not: 3 0 1 1 11

Survey Results 3. Would you like to have plus levels incorporated into a Benchmark Test? Definitely yes: Probably yes: Perhaps: Most likely not: Definitely not: 5 5 2 2 2

Conclusions • A “benchmark” test would be welcomed by most countries. • (The scores should be advisory in nature. ) • Providing “plus” level ratings would allow greater fidelity in making comparisons. • BILC should proceed with plans to: – Develop a benchmark STANAG test of reading comprehension. – Explore internet delivery options.

The Role of Proficiency Standards Dr. Martha Herzog BILC Athens, Greece 22 -26 June 2008

TRAINING IN TEST DESIGN • LANGUAGE TESTING SEMINAR – 20 Iterations – 265 participants – 38 nations – 4 NATO officers – Facilitators from 10 nations

BENCHMARK TESTS • Tests of all four skills • Measures Level 1 through Level 3

STANDARDS • All standards have three components – Content – Tasks – Accuracy

TEAMWORK • The Working Group functions as a team – 13 members from 8 nations – Contributions from many other nations

Summary • STANDARDS • TRAINING IN TEST DESIGN • TEAMWORK • TECHNOLOGY

Why does the BAT follow a construct-based, evidencecentered test design?

Why does the BAT follow a construct-based, evidence-centered test design? Because a CBT, ECD design solves a major problem encountered when testing proficiency in the receptive skills, i. e. in testing Reading and Listening.

In contrast to traditional test development procedures, CBT allows direct (rather than indirect application) of the STANAG 6001 Proficiency Scales to the development and scoring of Reading and Listening Proficiency Tests.

Test Development Procedures: Norm-Referenced Tests • • Create a table of test specifications. Train item writers in item-writing techniques. Develop items. Test the items for difficulty, discrimination, and reliability by administering them to several hundred learners. • Use statistics to eliminate “bad” items. • Administer the resulting test. • Report results compared to other students.

Test Development Procedures: Norm-Referenced Tests (cont. ) • Each test administration yields a total score. • However, setting “cut” scores or “passing” scores on norm-referenced tests is a major challenge. • And relating scores on norm-referenced tests to a polytomous set of criteria (such as levels in the STANAG 6001 or other proficiency scales) is even more problematic.

Test to be calibrated 100 50 0 Level 3 Group Level 2 Group Level 1 Groups of ”known” ability A Traditional Method Of Setting Cut Scores

Distinct “cut” scores between the scores of the calibration groups 100 50 0 Level 2 Group Level 1 Groups of “known” ability The Results One Hopes For: Level 3 Group

100 Bands of Overlapping Test Scores Level 3 Group ? ? ? 50 ? ? ? 0 Level 2 Group Level 1 Groups of ”known” ability The Results One Always Gets:

Where in the overlapping range should the cut score be set? 100 ? ? ? 50 ? ? ? 0 Level 3 Group Level 2 Group Level 1 Groups of ”known” ability No matter where the cut scores are set, they are wrong for someone.

Why is this “overlap” in scores always present? • A single test score on a multi-level test… – Gives equal credit for every right answer regardless of its proficiency level. – Camouflages by-level abilities. – Is a “compensatory” score. • Proficiency abilities… – Are by definition “non-compensatory”. – Require demonstration of sustained ability at each level.

A Better Test Design: Construct-Based Proficiency Testing • Uses a “floor” and “ceiling” approach similar to that used in Speaking and Writing tests. • The proficiency rating is assigned based on two separate scores: – A “floor” proficiency level of sustained ability across a range of tasks and contexts specific to that level. – A “ceiling” proficiency level of non-sustained ability at the next higher proficiency level.

Therefore Construct-Based Testing • Tests each proficiency level separately. – Three tests for levels 1 through 3. – Or three subtests within a longer test. • Rates each level-specific test separately. • Applies the “floor” and “ceiling” criteria used in rating productive skills using a scale such as: – – Sustained (consistent evidence) = 70% to 100% Developing (present, inconsistent)= 55% to 65% Emerging (some limited evidence) = 40% to 50% Random (no visible evidence) = 0% to 35%

Does it make a difference? Consider the following example.

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Level 1 Level 2 Level 3 Total Results Score True Level Alice 195 ? Bob 195 ? Carol 195 ?

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice 85 70 40 Total Score 195 Bob 195 Carol 195 True Level

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice 85 70 40 Sustained Emerging Total Score 195 Bob 195 Carol 195 True Level

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice 85 70 40 Sustained Emerging Total Score True Level 195 2 (Barely) Bob 195 Carol 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Total Score True Level 195 2 (Barely) 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random Total Score True Level 195 2 (Barely) 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random Total Score True Level 195 2 (Barely) 195 2 (Clearly) 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random 90 60 45 Total Score True Level 195 2 (Barely) 195 2 (Clearly) 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random 90 60 45 Sustained Developing Emerging Total Score True Level 195 2 (Barely) 195 2 (Clearly) 195

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Level 2 Level 3 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random 90 60 45 Sustained Developing Emerging Total Score True Level 195 2 (Barely) 195 2 (Clearly) 195 1 (Clearly)

A Total Score (where 195=Level 1) Versus Construct-Based Scoring Learner Results @ Level 1 Alice Bob Carol 85 70 40 Sustained Emerging 90 85 20 Sustained Random 90 60 45 Sustained Developing Emerging Total Score True Level 195 2 (Barely) 195 2 (Clearly) 195 1 (+ developing ability @ 2)

Scores on Construct-Based Tests are: valid, easily explained, and informative! But how is a CBT developed?

Test Development Procedures: Construct-Based Proficiency Tests 1. Define each proficiency level as a construct to be tested. 2. Follow a construct-based, evidencecentered test design. 3. Train item writers – In the proficiency scales. – In matching text types to the tasks in the scales. – In item writing.

Test Development Procedures: Construct-Based Proficiency Tests 4. Develop items that exactly match all of the specifications for each level in the proficiency scale, with. . . – Examinee task aligned with the author’s [or the speaker’s ] purpose. – Level-appropriate topics and contexts.

Test Development Procedures: Construct-Based Proficiency Tests 5. Use “alignment”, “bracketing”, and “modified Angoff” item review and quality control procedures. – A specifications review to insure alignment of author purpose, text type, and reader task. – A bracketing review to check the adequacy of the item’s response options for test takers at higher and at lower proficiency levels. – Modified Angoff ratings of item difficulty for “at -level” test takers to set passing levels.

Test Development Procedures: Construct-Based Proficiency Tests 6. Use data from the Angoff reviews to define “sustained ability” for each level of the test. 7. Assemble the “good” items into levelspecific tests or subtests. 8. Do validation testing. 9. Use statistical analyses to confirm reviewer ratings.

Test Development Procedures: Construct-Based Proficiency Tests 10. Replace items that do not “cluster” or act like the other items at each level. 11. Score and report results for each level using “sustained” proficiency criteria. 12. Continue to build the item data bases to enable: – Random selection of test items for multiple forms. – Computer adaptive testing.

What do the results of a CBT Reading proficiency look like? Here are some initial results from the BAT English Reading Proficiency Test.

Results on the Level 1 Test Results on the Level 2 Test Results on the Level 3 Test

When will BAT be available? • Funds have been set aside for administering and scoring of 200 free advisory tests. – All four skills will be tested. – BAT Reading, Listening, and Writing tests will be online tests. – The Speaking test will be conducted over the telephone. • These tests are to be used in test norming or calibration studies.

When will BAT be available? • We anticipate the following timeline: – About October, 2008. Directions on how to apply will be sent out. – About November, 2008. Applications will be submitted. – About December, 2008. Applications will be reviewed and decisions made about how the 200 tests will be allocated. – Between February and June. The first round of advisory testing will be conducted.

When will BAT be available? • More specific information will be sent out after consultation with ACT.

Are there any questions? ? ? ?