PILOTING CROATIAN EXPERIENCE Tamara Kramari Maras However well

PILOTING CROATIAN EXPERIENCE Tamara Kramarić Maras

However well designed an examination may be, and however carefully it has been edited, it is not possible to know how it will work until it has been tried out on students. Alderson, Clapham and Wall. 1995. Language Test Construction and Evaluation, Cambridge: University Press

test design and administration planning challenges control group interpreting the statistics

CHALLENGES • establishing a system • number of pilot tests and items • piloting tests or sets of items • pre-testing before piloting • finding the suitable population (size, time) • administering the test • interpreting the statistics • establishing the control group

PLANNING • how many pilot tests a year? • where? • population size and level? • the most appropriate dates? • budget?

• 2 pilot tests a year • Slovenia , Bi. H • May/November • 35 -60, L 1+, L 2, L 3 • bilateral activity, host country responsible for costs of accomodation/food • a detailed rationale

PILOT TEST CONSTRUCTION AND ADMINISTRATION • what should the pilot test look like? • more items than in the real test? • pre-piloting before the pilot? • test administration • student preparation and motivation

• resembling the real test as much as possible (number of tasks/items, layout, task types, rubrics, time) • pre-piloting on available population in FLS • administering the test ourselves + standardisation session • students need to understand the mutual benefits

INTERPRETATION OF STATISTICS • satisfactory values of FV/DI? • ALCPT as a factor • anchor items

• L 2 FV 70% - 90% DI 0, 2 -0, 4 • L 3 FV 40% - 60% DI 0, 5 -1 (bi-level tests of reading/listening) (population of 1/3 L 3, 2/3 L 1+, L 2)

CONTROL GROUP • not enough candidates in FLS for L 2 /L 3 piloting in Croatia • who else could be involved in piloting?

• establishing a control group • volunteers (L 2/L 3 or ALCPT 80+), cadets • importance of motivation • updating and extending the control group

BENEFITS OF PILOTING • main validation process • standardisation through moderation • standardisation sessions (oral and written tests) • new ideas • backwash on teaching/learning