Designing and Developing Useful Language Tests Lyle F

Designing and Developing Useful Language Tests Lyle F. Bachman Department of Applied Linguistics & TESL University of California, Los Angeles

Uses of Language Tests F to make inferences about individuals’ language ability F to make predictions about individuals’ ability to use language in contexts outside the test itself F to make decisions about individuals

Correspondences between Language Test Performance and Language Use

Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Characteristics of the Language User Characteristics of the Test Taker

Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Language Ability of the Language User Language Ability of the Test Taker

Characteristics of the Language Use Task Characteristics of the Language Test Task Language Use Language Test Performance Language Ability of the Language User Language Ability of the Test Taker

A Framework of Task Characteristics Language Use Task an activity that involves individuals in using language for the purpose of achieving a particular goal in a particular situation (Bachman & Palmer, 1996, p. 44)

A Framework of Task Characteristics Target Language Use (TLU) Domain a set of specific language use tasks that the test taker is likely to encounter outside of the test itself, and to which we want our inferences about language ability to extrapolate (Bachman & Palmer, 1996, p. 44)

A Framework of Task Characteristics Ø Setting: the physical circumstances under which either language use or language testing takes place. Includes the physical characteristics of the setting, the participants involved, and the time of the task.

A Framework of Task Characteristics Ø Rubric: the context for the task; includes those characteristics that provide the structure for the task, and constrain how language users or test takers are expected to respond to these tasks

A Framework of Task Characteristics Ø Input: material contained in the task, which test takers need to process in some way, and to which they are expected to respond.

A Framework of Task Characteristics Ø Expected response: the language use that is expected, given the way in which the rubric, or context, for the task is configured, and the particular input that is provided

A Framework of Task Characteristics Ø Relationship between input and response: • • • Reactivity: the degree of reciprocity, or interaction involved, Scope: the amount and range of input that needs to be processed in order to respond, and Directness: the extent to which the response can be made by using information in the input by itself, or whether the language user or test taker must also rely on information in the context or in his or her own real world knowledge.

Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe TLU tasks in terms of their task characteristics.

TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

TLU Task Type A (letter) TLU Task Type B (memo) Modify TLU Task Type to Develop Test Task Specifications A (write formal letter to customer) Test Task Specifications B (write memo to co-worker)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

Example Test Development Project setting: telephone company needs to hire new employees

Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

Example TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing

Selecting TLU Tasks for Use in Designing Test Tasks F Some TLU tasks may be essential to performing the job

Example TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing

Selecting TLU Tasks for Use in Designing Test Tasks F Some TLU tasks may not require the areas of language ability we want to measure. F Some TLU tasks may not be appropriate for all test takers.

Describing Target Language Use Tasks 1. Identify the TLU tasks in the relevant domain. 2. Select TLU tasks as a basis for designing test tasks. 3. Describe these tasks in terms of their task characteristics.

TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 2 phone discussions with clients, co-workers Task 4 responding to customer complaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo)

Developing Test Tasks F Modify TLU task types for use as test tasks. F Create original test tasks.

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

TLU Task Type A (letter) TLU Task Type B (memo) Modify TLU Task Type to Develop Test Task Specifications A (write formal letter to customer) Test Task Specifications B (write memo to co-worker)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

Conclusion Bachman-Palmer Approach to Language Test Development F Provides test developers with a theoretically grounded and principled basis for developing and using language tests.

Conclusion Bachman-Palmer Approach to Language Test Development F Provides test developers with an understanding that will enable them to make their own decisions about either selecting or developing a language test that will be useful for their particular situation.

Conclusion Bachman-Palmer Approach to Language Test Development F Requires test developers to consider the fundamental correspondence between language test performance and non-test language use.