As slippery as an eel Assessing speaking and

As slippery as an eel? Assessing speaking and writing Part One Ülle Türk University of Tartu Estonian Defence Forces 23 rd CSW, Tampere, 27 -29 March 2009

Testing writing? l l l l l Fill in the gaps with suitable words so that the text is true for you. Re-write the text in the future tense. Fill in the form with the information given in the box. Read the letter and write an answer. Write an essay on the topic “Why study English? ” Study the pictures, put them in the order you think best and write the story. Read the text and write a short summary of it. You bought a new dictionary yesterday, but found later that several pages were missing. Write a letter to the manager of the shop informing him of the problem and telling him what you want him to do about it. Read the basic facts about Australian history and then write a short report. 2

Questions l l What is it exactly that we assess when we say we assess students’ speaking and writing skills? How do we arrive at a common understanding of what is ‘good’ writing, what is a ‘good’ oral presentation or what constitutes ‘good’ spoken or written communication? 3

Terms l Assessment l l Formal informal Continuous fixed-point Formative summative Testing l l l Achievement Proficiency Diagnostic Placement High-stakes low-stakes 4

Assessment/ test quality l l l Validity Reliability Authenticity Washback Practicality 5

Validity: definitions l l A good test needs to be valid. = It must test what it is meant to test. an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores. § S. A. Messick (“Validity” in R. L. Linn (ed. ) Educational Measurement. 1989, p. 13) 6

Validity l l Does the test match the curriculum, or its specifications? Is the test based adequately on a relevant and acceptable theory? Does the test yield results similar to those from a test known to be valid for the same audience and purpose? Does the test predict a learner’s future achievements? 7

Validity l l l Content validity Construct validity Criterion-related validity Predictive validity Construct validity is indeed the unifying concept that integrates criterion and content considerations into a common framework for testing rational hypotheses about theoretically relevant relationships. § Messick, S. A. “Test validity and the ethics of assessment. ” American Psychologist 35, 1980, p. 1015 8

Threats to test validity l l construct irrelevant variance construct under-representation 9

Factors affecting validity l l l Lack of specifications Lack of training of item/ test writers Lack of / unclear criteria for marking Lack of piloting/ pre-testing Lack of detailed analysis of items/ tasks Lack of feedback to candidates and teachers 10

Communicative competence 1 l Canale & Swain (1980), Bachman (1990): language knowledge types l l l Linguistic knowledge Discourse knowledge Sociolinguistic knowledge 11

Grabe & Kaplan (1996): Model of Writing l Components of language knowledge relevant to writing l l linguistic knowledge: written code, morphology, vocabulary, syntax discourse knowledge: cohesion, structure, genre sociolinguistic knowledge: functional uses of writing, register, situational parameters Influential in teaching and testing of writing (e. g. , Weigle, 2002) 12

Communicative competence 2 l Bachman & Palmer (1996): communicative language ability l l Language knowledge Strategic competence 13

Douglas (2000): Specific Purpose Language Ability l Language knowledge l l grammatical knowledge textual knowledge l l l functional knowledge sociolinguistic knowledge Strategic competence l l l rhetorical organization cohesion assessment goal setting planning control of execution Background knowledge l discourse domain 14

Common European Framework of Reference for Languages: Learning, teaching, assessment (2001) l l General competences Communicative language competences 15

General competences Declarative knowledge of the world sociocultural knowledge Skills and know -how ‘Existential’ competence Ability to learn practical attitudes language skills (social, living, vocational/ professional, leisure skills) awareness motivation values beliefs cognitive intercultural skills and communicative awareness styles personality factors general phonetic awareness and skills study skills heuristic skills 16

Communicative language competences Linguistic competences lexical competence Sociolinguistic competences linguistic markers of social Pragmatic competences discourse competence relations grammatical politeness conventions functional competence semantic expressions of folk wisdom competence phonological register differences competence orthographic dialect and accent competence orthoepic competence 17

Communicative language activities and strategies l l l productive activities and strategies receptive activities and strategies interactive activities and strategies mediating activities and strategies non-verbal communication l l l practical actions paralinguistics paratextual features 18

Oral production l l public address (information, instructions, etc. ) addressing audiences (speeches at public meetings, university lectures, sermons, entertainment, sports commentaries, sales presentations, etc. ) l l l reading a written text aloud speaking from notes, or from a written text, or from visual aids acting out a rehearsed role speaking spontaneously singing 19

Spoken interaction l l l l l transactions; casual conversation; informal discussion; formal discussion, debate; interview, negotiation; co-planning; practical goal-oriented co-operation. 20

Oral mediation l l l simultaneous interpretation (conferences, meetings, formal speeches, etc. ) consecutive interpretation (speeches of welcome, guided tours, etc. ) informal interpretation: l l l of foreign visitors in own country of native speakers when abroad in social and transactional situations for friends family, clients, foreign guests, etc. of signs, menus, notices, etc. 21

CEFR levels The Common European Framework of Reference (Council of Europe 2001) defines communicative proficiency l At six levels, arranged in three bands A 1 A 2 B 1 B 2 C 1 C 2 l in relation to six skills: listening, reading, spoken interaction, spoken production, written interaction, written production l in the form of “can do” statements 22

Getting to know the levels l The self-assessment grid is not enough l More specific scales: l l l CEFR Ch 4: descriptors of communicative activities l CEFR Ch 5: descriptors of linguistic competence The ELP (European Language Portfolio) Manual: Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR) 23

I can deal with most situations likely to arise whilst travelling in an area where the language is spoken. I can enter unprepared into conversation on topics that are familiar, of personal interest or pertinent to everyday life (e. g. family, hobbies, work, travel and current events). Self-assessment grid (CEFR and standard adult passport) 24

Cercle. S ELP: goal-setting and selfassessment checklists 25

Questions to ask l What competences should my students have in l l l Spoken interaction Spoken production Written interaction Written production What tasks should they be able to perform to demonstrate their mastery of the competences? How well should they be able to perform them? 26

Reliability l l A test needs to be reliable. = It must produce consistent results at different times. NB! A test that is not reliable cannot, by definition, be valid. 27

Reliability l l l If I take the test again tomorrow, will I get the same result? If I take a different version of the test, will I get the same result? If the test had different items, would I have got the same result? Do all markers agree on the mark I got? If the same marker marks my test paper again tomorrow, will I get the same result? 28

Factors affecting reliability l l l Poor administration conditions – noise, lighting, cheating Lack of information beforehand Lack of specifications Lack of marker training Lack of standardisation Lack of monitoring 29

References l l Bachman, Lyle F. (1990) Fundamental Considerations in Language Testing, Oxford University Press, Oxford. Bachman, Lyle F. and Palmer, Adrian (1996) Language Testing in Practice, Oxford University Press, Oxford. Cushing Weigle, Sara (2002) Assessing Writing, Cambridge University Press, Cambridge. Douglas, Dan (2000) Assessing Languages for Specific Purposes, Cambridge University Press, Cambridge. 30