STANAG 6001 Speaking Test Observations Lessons learned Tamar

STANAG 6001 Speaking Test Observations; Lessons learned Tamar Shavlakadze BILC Professional Seminar Tbilisi, Georgia 1 -5 October 2017 1

General Outline of the Presentation v Speaking test format (the core of the test and tester stance) v Global tasks and communicative functions (macro and micro functions) v Evaluation (Fluency & Accuracy) 2

Who are the test takers? v Resolute Support Mission; v Promotion to higher ranks; v International positions abroad; v Courses abroad; v Graduates of English language courses of the Language Training School of NDA; v Graduates of Bachelors Programme of National Defence Academy; v. Applicants for the positions of teachers and translators. 3

General Test Format �Bi-level (levels 1 -2; 2 -3) �Length: Bi-level 1 -2 20 -25 minutes Bi-level 2 -3 30 -35 minutes �Two examiners (interlocutor/rater) �Guided interview �Three mandatory phases 4

Core of the test and three Perspectives Level Checks Probes Three The core Perspectives Psychological Proves the examinee what he she or he can do or she cannot do Linguistic Checks for tasks and content which examinee performs with greatest least accuracy Evaluative Finds the examinee’s Finds level at which examinees speaking level can no longer speak accurately 5

Tester stance �TTT (Teachers/testers) talking time must be at a minimum in both cases; �Lessons as well as speaking tests should be contextualized; �Instruction checking questions should be used; �Teachers/testers should create friendly atmosphere; �Teachers/testers should be in control of the lesson/exam; �Pursue topics with follow-up questions. 6

Tester Stance Testers Should NOT: ØCorrect examinees; ØFinish their sentence; ØUse body language; ØFeed vocabulary item; ØGo informal or start chatting with candidates. 7

Macro function – categories for the functional use of spoken and written discourse consisting of a sequence of sentences. Factually oriented talk Evaluative talk description explanation narration justification instruction prediction comparison decision Bygate (1987) 8

Micro functions are related to individual actions completed within a turn in an interaction. CEF groups them into 6 main categories: �giving and asking for factual information, e. g. identifying, reporting, correcting, asking, answering; �expressing and asking about attitudes, e. g. agreement/disagreement, knowledge/ignorance, ability, intentions, preference, emotions, apologies, sympathy. �suasion, e. g. suggestions, requesting, warnings, advice, asking help, invitations, offers; �socializing, e. g. attracting attention, addressing, greetings, introduction; ; �structuring discourse, e. g. opening, turn taking, closing, changing theme; �communication repair, e. g. signaling nonunderstanding, appealing for assistance, paraphrasing. 9

Global tasks and Communicative Functions 0 NO PROFICIENCY Level 1 SURVIVAL Level 2 FUNCTIONAL Level 3 PROFESSIONAL Short conversation Ask and answer simple questions (present tense) Handle a simple transaction (RP) Short Narration in present Short description Narration ( Present, Past, future) Description (person, place, object) Report/state facts on a current event Instructions /directions Compare/contrast Information Gathering Task (in past tense) Deal with a familiar situation with a complication (RP) Supported opinions Hypothesize Discussing an abstract topic State and defend policy 10

Holistic Assessment Scale Levels & Tasks Accuracy Statements 0 NO PROFICIENCY 0+ MEMORIZED PROFICIENCY Three level checks at level 1 1 SURVIVAL Short conversation Ask and answer simple questions Handle a simple transaction (RP) Short Narration Short Description 1+ SURVIVAL+ Three probes at Level 2 (RP) No functional ability None or occasional isolated words Text produced Random words & phrases Memorized words and short phrases Can ask questions or make statements only with memorized material and set expressions Most utterances are telegraphic Linking words and markers are omitted, confused, or distorted Communication is severely limited Faulty pronunciation, stress and intonation Can create sentences Discrete Can begin, maintain, and close short conversations sentences Can satisfy simple, predictable, personal and accommodation needs Can meet minimum courtesy, introduction, and identification requirements Communicate about routine tasks in the workplace Seldom speaks with natural fluency, and cannot produce continuous discourse, except with rehearsed material Can speak at the sentence level and may produce strings of two or more simple sentences joined by common linking words Frequent errors in pronunciation, vocabulary, and grammar often distort meaning May only use one tense or tend to avoid certain structures Speech is often characterized by hesitations, erratic word order, groping for words Ineffective reformation and self-corrections Can easily take part in short conversations by asking and answering simple questions Shows a very limited and inconsistent ability to handle longer conversations on concrete topics Discourse consists of strings of related sentences but not full paragraphs Can satisfy a few social demands and provide more than skeletal information when supplying biographical background Can readily ask for assistance; request information and clarification; and express satisfaction, dissatisfaction, and confirmation May hesitate and even have to change the subject because of lack of language resources Simple structures and basic grammatical relations are only somewhat controlled Time references may be used incorrectly Vocabulary may be imprecise except for the highest frequency utterances Frequent errors in pronunciation, vocabulary, or grammar may impede communication Delivery may be labored Strings of related sentences 11

2 FUNCTIONAL Past narration Description Report on a current event Instructions /directions Deal with a familiar situation with a complication RP IGT in Past tense 2+ FUNCTIONAL+ Three probes at level 3 3 PROFESSIONAL Supported opinions Hypothesizing Discuss an abstract topic State and defend Policy Clarify points Can narrate current, past and future activities Can describe people, places, and things Can create full paragraphs, minimally cohesive Can confidently handle most normal, casual conversations on concrete topics Can often elaborate in common daily communicative situations Can give complicated and detailed directions and make non-routine changes in travel and other arrangements Simple structures and basic grammatical relations are typically controlled, while more complex structures are used inaccurately or avoided Vocabulary use is appropriate for high-frequency utterances but unusual and imprecise at other times Errors in pronunciation, vocabulary, and grammar may sometimes distort meaning Generally speaks in a way that is appropriate to the situation, although command of the spoken language is not always firm Full paragraphs, minimally cohesive Can create a significant amount of discourse beyond the paragraph level Can use the language effectively to narrate, describe, state facts, compare and contrast, give detailed instructions and directions Can use the language with less ease and effectiveness to support opinion, clarify points, and answer objections Shows some linguistic limitations when dealing with unfamiliar subjects and situations Can generally elicit information and informed opinion Can often discuss abstract concepts but can rarely use abstract linguistic formulations successfully Errors in vocabulary and more complex structures and cohesive features sometimes interfere with efforts to elaborate on an argument or point of view Pronunciation errors may occasionally impede communication Speech is usually appropriate to the situation Discourse beyond the paragraph level Produces extended discourse and conveys meaning correctly and effectively Use of structural devices is flexible and elaborate Speaks readily and in a way that is appropriate to the situation Without searching for words or phrases, can use the language clearly and relatively naturally to elaborate on concepts freely Can convey abstract concepts Can easily repair the conversation Pronunciation may be foreign Errors may occur in low frequency or highly complex structures characteristic of a formal style of speech Extended discourse 12

Fluency & Accuracy - Opposite ends of a continuum in which at extremes speech is seen as accurate and disfluent or inaccurate and fluent �Two components of assessment - Accuracy of structure and vocabulary - Quality and speed of delivery 13

‘Error gravity’ – determines seriousness of errors �- John buy a newspaper every day. � - Every day a newspaper buy. (James, 1998) 14

Definitions of fluency Fluency markers �flow of smoothness �rate of speech � absence of excessive pausing � absence of disturbing hesitation markers � length of utterances �connectedness 15

Central part of fluency �Speaking rate �Speech –pause relationships �Frequency of dysfluency markers ( hesitations, repetitions, self-corrections) �“Small words” and generic words 16

Range of Vocabulary Grammar Structures Level 2 High frequency concrete Voceyeglasses; mast; etc. Simple structures Level 3 Low frequency concrete voc Complex grammatical spectacles structures High frequency abstract Voc obligation; reduction; permanent; dilemma Low frequency concrete voc – Low frequency monocle; mizzenmast ; camshaft grammatical structures Level 4 Low frequency abstractconundrum; scoff; abhor; rebuke. 17

Bibliography �Testing Second Language by Glenn Fulcher �Assessing Speaking by Sari Luoma �Common European Framework of Reference for Languages: Learning, teaching, assessment �Standardization Agreement 6001 (5 th addition ) �LTS/ALTS material (Trisection of speaking and OPI checklist) 18

Thank you 19