Hellenic Ministry of Culture Education and Religious Affairs

Hellenic Ministry of Culture, Education and Religious Affairs National & Kapodistrian University of Athens, Aristotle University of Thessaloniki & Computer Technology Institute of the University of Patras Multilingual Platform for CALT (Computer Adaptive Language Testing) in six languages The National Foreign Language Exams for the State Certificate of Foreign Language Proficiency in Greece: The KPG examination suite http: //rcel. enl. uoa. gr/kpg Professor Bessie Dendrinos, President of the KPG Examination Board 02/10/2020 1

02/10/2020 2 National Strategic Reference Framework (NSRF) – 2007 -2013

e-KPG includes �The multilingual CALT platform �Platform for the e-training of oral examiners and script raters (http: //galaxy. cti. gr/e-kpgdevel 23/) �The KPG e-school (http: //rcel. enl. uoa. gr/kpgeschool/) 02/10/2020 3

Multilingual CALT platform English German French Italian Spanish Turkish 02/10/2020 4

Multilingual standards �Both the computer-based and the paper- based KPG exams contain common: proficiency scaling (aligned to the 6 level scale of the Council of Europe) illustrative descriptors (aligned to the CEFR) accompanying criterial features (constructivist) view of language-in-use 02/10/2020 5

Multilingual framework �The exam suite is based on a generic (portable across languages) methodological framework suitable for describing language performance across proficiency levels. 02/10/2020 6

Multilingual operationalization �And… across languages, the exams: test common competences and strategies are regulated by common specifications use a common test-task typology 02/10/2020 7

Basic tools of the CALT platform Authoring tool Piloting tool Test player 02/10/2020 8

Authoring tool roles Diagram showing the flow of roles, and work required to insert test tasks and give final approval 02/10/2020 9

Testing understanding �The KPG CALT is developed to test both comprehension and production: �The first part tests: reading comprehension listening comprehension language awareness 02/10/2020 10

The task-based CALT �The CALT is not item but task based. �The smallest unit of the comprehension part is a task, not an item. �Three test tasks comprise a ‘testlet’. �The candidate’s performance in each testlet determines what testlet level s/he will be served next. 02/10/2020 11

Composition of testlets Each testlet is composed of three tasks: 1) One reading comprehension task 2) One language awareness task 3) One listening comprehension task There is a common (across languages) formula as to the mix of the types of tasks that each testlet contains. 02/10/2020 12

CALT task typology � The KPG CALT contains 10 task types giving us in all 12 types of testlets. � Each task is composed of 5 items as follows: a) items testing comprehension of a source (written or audio) text b) items testing awareness of correct and appropriate use of language 02/10/2020 13

CALT testlet typology � 3 tasks comprise a testlet � 3 testlets comprise a testing unit � 3 -5 testing units comprise a complete test for each of the lower levels (Α 1 -Β 1) contains 3 testlets (45 items in total) a complete test for each of the higher levels (B 2 -C 2 level) contains 4 testlets (60 items). 02/10/2020 14

Starting the test and moving through it �The candidate chooses the level of the testlet s/he wants to start with and then if s/he is successful on 50% of the items, the next testlet is of the same level his/her success is below 50%, the next testlet is of lower level If his/her success is above 50%, the next testlet is of higher level 02/10/2020 15

Digital e-test typology 02/10/2020 16

Reading comprehension (French B 2) Multiple choice (MC)-tick 02/10/2020 17

Reading comprehension (German C 1) Multiple choice (MC)-tick 02/10/2020 18

Reading comprehension (English C 2) Multiple choice (MC)-tick 02/10/2020 19

Language awareness (Italian C 1) Multiple match-insert missing item 02/10/2020 20

Language awareness (Spanish A 2) Matching-drag-’n-drop (Q&A) 02/10/2020 21

Language awareness (English B 1) Fill in - write in a word 02/10/2020 22

Language awarness (German A 1) Match-drag’n-drop 02/10/2020 23

Language awareness (English B 1) writing in a word 02/10/2020 24

Listening comprehension (A 1) Multiple match (drag’n-drop letter/number) 02/10/2020 25

Listening comprehension (Italian B 1) Multiple match (drag-’n-drop letter/number) 02/10/2020 26

Testing production �Production performance is also computer based �The tests on reading and listening comprehension determine the level of the speaking and writing tests in which candidates will be asked to participate 02/10/2020 27

Testing leveled performance TESTING SPEAKING & ORAL MEDIATION 02/10/2020 TESTING WRITING & WRITTEN MEDIATION 28

Oral production �Similar test tasks across languages are aimed to assess context specific: oral performance, stimulated by visual or multimodal texts in the target language oral mediation performance, stimulated by texts in the home language 02/10/2020 29

Speaking task (German B 1) Diese Fotos stammen aus Ihrem elektronischen Familienalbum. Welche Personen sind auf den Fotos abgebildet? Wann und warum wurde jedes einzelne Foto gemacht? 02/10/2020 30

Speaking task (Italian B 2) A tuo parere, prima della ripresa di queste due fotografie, che cosa può essere accaduto? E cosa hanno in comune i due avvenimenti? 02/10/2020 31

Speaking task (Spanish B 2) Dinos cómo crees que se sienten estas personas y por qué. 02/10/2020 32

Mediation in English (C 2) Tell your friend Edward about two games he knows almost nothing about: backgammon and jenga. Click here to speak 02/10/2020 33

Writing production �Similar test tasks across languages are aimed to assess context specific: writing performance, stimulated by cues, visual or multimodal texts in the target language written mediation performance, stimulated by texts in the home language 02/10/2020 34

Writing task (French B 2) 02/10/2020 35

Writing task (Spanish A 2) Escribe un e-mail (60 -70 palabras) a tu amiga Valentina para hablarle de un bonito lugar de tu ciudad donde estuviste ayer. Cuéntale: 1. Qué tipo de lugar era 2. Dónde está exactamente 3. Qué hiciste allí 02/10/2020 36

Writing task (English A 1) 02/10/2020 37

Written mediation in Italian (C 1) 02/10/2020 38

Test task piloting Entypo 1 - B 1 100 80 d. i. *100 60 FR 40 DE UK 20 IT Poly. (FR) 0 0 10 20 30 40 50 60 70 80 90 100 110 Poly. (DE) Poly. (UK) Poly. (IT) -20 e. i. *100 02/10/2020 39

The Piloting Tool_1 �It’s a management system linked to the Authoring Tool. �Its purpose is to manage trial runs (or the piloting) of each test task, in real-time conditions, with a view to examining the task’s validity. 02/10/2020 40

How does the PT operate? �Groups of tasks are packed in groups and piloted by sample population of candidates �Depending on how well the tasks have ‘behaved’, they may be considered: good enough to be inserted in the Test Bank partially problematic and in need of amendments seriously problematic (warranting serious modification or ditching) 02/10/2020 41

The piloting tool_2 �The PT has been designed to measure basic indices with an inbuilt statistics system �The functions of the PT are available to the members of the language-specific author team who have been assigned the role of ‘task evaluator’ and/or the role of ‘Administrator’. 02/10/2020 42

Managing the test packs They are managed as follows: The user ‘characterizes’ a total number of tasks on the basis of the task. typology and creates test packages Token are created for the sample population who will use them. Indices of statistical interest are worked out for results and ultimately findings. 02/10/2020 43

www. rcel. enl. uoa. gr 02/10/2020 44