Uneven language proficiency How spiky can a spiky

Uneven language proficiency: How spiky can a spiky profile be? Karen Smith Lecturer EAP/Study Skills University of Central Lancashire, Preston

Overview • Research questions: • What is the shape of uneven proficiency in EAP candidates across the 4 skills? • Are there correlations between the skills? • What are ‘spiky profiles’ and how ‘spiky’ can they get?

‘spikiness’ Should we expect big score differences across the skills? How much variation is possible or natural? Should we expect some skills to be always stronger than others? Does variation matter?

Are the 4 skills connected? • In L 1 listening and speaking develop together • Then ‘literate’ skills grow from speaking & listening • Writing skills grow from speaking & listening (Berninger & Abbot, 2010) • The same brain mechanism is used for syntactic encoding in speaking and writing (Cleland & Pickering, 2006) • Speaking does not always precede writing in L 2 • Individual differences have an impact: modality strength, preference, cognitive style, affective factors (Weissberg, 2006; Kormos & Trebits 2012; Baba, Takemoto & Yokochi 2013)

UCLan’s TELL • Listening (30 mins) 3 different audio recordings, diff types of interaction: informal, semi-formal and formal • 20 marks • Reading (55 mins) range of texts, diff. skills tested • 30 marks • Writing (45 mins) essay writing task of 180 -200 words (B 2) 250 -280 (C 1) • 20 marks (10 bands) • Listening & reading marked optically • Min 20% of all scripts 2 nd marked • Speaking • 11 mins (B 2) 13 mins (C 1) • 3 parts: intro, interactive discussion, responding to questions • Grammar, vocab, pronunciation, discourse management, interactive ability • 30 marks (5 bands)

Exam Data from the TELL • B 2 & C 1 levels under investigation • December 2016 sittings in 4 centres: Preston, Sunderland, Hull & Coventry • 542 B 2 candidates’ l/r/w/s scores compared • 56 C 1 candidates’ l/r/w/s scores compared • Mixed nationalities, all adults, 21 -25 years old

One-upmanship among the skills? B 2 data n=542 C 1 data n=56 Speaking better than Writing Reading Listening 96. 9% (0. 7% same) 100% 79. 5% (2% same) 84% (7. 1% same) 70. 8% (2. 4% same) 71. 4% (1. 8% same) Listening better than 84. 5% (7. 2% same) 96. 4% (1. 8% same) 63. 5% (3. 3% same) 78. 6% Reading better than 76. 9% (2. 4% same) 94. 6% (1. 8% same) Writing better than 20. 7% (2. 4% same) 3. 6% (1. 8% same) Speaking 26. 8% (2. 4% same) 26. 8% (1. 8% same) 33. 2% (3. 3% same) 21. 4% 18. 5% (2% same) 8. 9% (7. 1% same) 8. 3% (7. 2% same) 1. 8% (1. 8% same) 2. 4% (0. 7% same) 0%

Spearman’s rho correlation of each skill with overall score Skills Type B 2 (n=542) rs Reading & Overall Receptive Literate . 890 (79%) L. 849 (72%) L Listening & Overall Receptive Innate . 860 (74%)L. 642 (41%) L Speaking & Overall Productive Innate . 797 (64%) L. 715 (51%) L Writing & Overall Productive Literate . 785 (62%) L. 841 (71%) L C 1 (n=56)

December 2016 Skills’ correlations at B 2 & C 1 Skills Type B 2 (n=542) R Coefficient of determination in brackets B 2 (n=542) rs C 1 (n=56) rs Listening & Reading Receptive . 698 (49%)L . 706 (50%)L . 316 (10%) M Reading & Writing Literate . 654 (43%) . 662 (44%) . 794 (63%) L Speaking & Listening Innate . 574 (33%) . 594 (35%) . 557 (31%) L Speaking & Reading . 556 (31%) . 571 (33%) . 459 (21%) M Listening & Writing . 524 (27%) . 546 (30%) . 282 (8%) S . 518 (27%) . 523 (27%) . 374 (14%) M Speaking & Writing Productive

Quartiles of the overall score at B 2 Overall score % of sample mean score Speaking mean score Listening mean score Reading mean score Writing S-W% L-W% gap R-W% gap S-R% gap S-L% gap L-R% gap 0 -25 0 26 -50 20. 3 59. 32 45. 41 37. 55 35. 77 23. 55 9. 64 1. 78 21. 77 13. 91 7. 86 51 -75 55. 5 75. 12 68. 55 61. 24 47. 76 27. 36 20. 79 13. 48 13. 88 6. 57 7. 31 76 -100 24. 2 92. 92 85. 76 84. 44 65. 99 26. 93 19. 77 18. 45 8. 48 7. 16 1. 32

A closer look at Writing vs. Speaking at B 2 Speaking of Mean % gap scores sample with writing n=542 score SD Writing scores % of sample n=542 Mean % gap with speaking score SD 0 -25 0. 2 35. 00 0 0 -25 3. 4 44. 11 14. 50 26 -50 3. 3 10. 22 6. 99 26 -50 61 29. 20 13. 66 51 -75 45. 2 21. 16 10. 62 51 -75 31 22. 91 12. 17 76 -100 51. 3 32. 97 14. 29 76 -100 4. 6 9. 80 6. 48

0 CEG 08035196 CEG 08028765 CEG 08035377 CEG 08032812 CEG 08036226 CEG 08035005 CEG 08033164 CEG 08032834 CEG 08035137 CEG 08036213 CEG 08032681 CEG 08035323 CEG 08036228 CEG 08035145 CEG 08032066 CEG 08033264 CEG 08035849 CEG 08033787 CEG 08033856 CEG 08035689 CEG 08026434 CEG 08032405 CEG 08034118 CEG 08034930 CEG 08035594 CEG 08036268 CEG 08034862 CEG 08035197 CEG 08036015 CEG 08033813 CEG 08035590 CEG 08035335 CEG 08036046 CEG 08036315 CEG 08033089 CEG 08034873 CEG 08035314 CEG 08035894 CEG 08031711 CEG 08034561 CEG 08034981 CEG 08035568 CEG 08033736 CEG 08034838 CEG 08034185 CEG 08033051 CEG 08033097 CEG 08035866 CEG 08031317 CEG 08033815 CEG 08034603 CEG 08035273 CEG 08030321 CEG 08033402 CEG 08034927 CEG 08035516 CEG 08036182 CEG 08034388 CEG 08035971 CEG 08035051 CEG 08035987 CEG 08032821 CEG 08034728 CEG 08035701 CEG 08034150 CEG 08033197 CEG 08035527 CEG 08036250 CEG 08035702 CEG 08034256 CEG 08035669 CEG 08036376 CEG 08034841 CEG 08035728 CEG 08030491 CEG 08035598 CEG 08036024 CEG 08033642 CEG 08036192 CEG 08035558 CEG 08034656 CEG 08031086 CEG 08029230 CEG 08035442 CEG 08036296 CEG 08036235 CEG 08035886 CEG 08035234 CEG 08036389 CEG 08034013 CEG 08035375 Percentage weighted score Writing vs. speaking scores B 2 December 2016 120 100 80 60 40 20 Series 1 Candidates n=542 Series 2

A closer look at Reading vs. Listening Reading scores of Mean % gap SD sample with listening n=542 score Listening of Mean % gap scores sample with reading n=542 score SD 0 -25 3% 26. 13 11. 92 0 -25 1% 9. 50 9. 00 26 -50 27% 14. 85 11. 08 26 -50 20% 10. 97 8. 69 51 -75 41% 12. 46 9. 25 51 -75 45% 12. 89 9. 47 76 -100 29% 8. 46 6. 18 76 -100 34% 12. 58 10. 52

The picture so far… • What are the parameters of uneven proficiency across the 4 skills? • • • Speaking appears to be the strongest, even in the weakest candidates Writing is the weakest link for all (no surprise there!) Writing is very rarely better than speaking (2. 4% of candidates) Reading also lets down the weakest Top quartile writing scores show smallest gap with speaking • Are there correlations between the skills? • receptive skills (R&L) are the most correlated at B 2, not C 1 • literate skills (R&W) also strongly correlated, more at C 1 • Despite both being productive, Speaking & Writing are least correlated

Further study in spikiness… • Compare with other proficiency test data • Closer inspection of the speaking and writing scores: what is going on within the scores? • Compare performance under exam conditions against relaxed conditions (affective dimension) • Explore modality preference & self-efficacy (affective/personality-related)

References Baba, K. , Takemoto, Y. and M. Yokochi. 2013. ‘Relationship between second language speaking and writing skills and modality preference of university EFL students’. Japanese Institutional Repositories Online. (Retrieved 14 March 2015 from http: //jairo. nii. ac. jp/0263/00000548. ) Bereiter, C. & M. Scardamalia. 1987. The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum Associates. Berninger, V. W. , & Abbott, R. D. 2010. ‘Listening comprehension, oral expression, reading comprehension, and written expression: Related yet unique language systems in grades 1, 3, 5, and 7’. Journal of Educational Psychology, 102/3: 635 -651. Cleland, A. and M. Pickering. 2006. ‘Do writing and speaking employ the same syntactic representations? ’. Journal of Memory and Language 54: 185 – 198. Council of Europe. 2011. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Language Policy Unit, Strasbourg. (Retrieved 14 March 2015 from http: //www. coe. int/t/dg 4/linguistic/Source/Framework_EN. pdf. ) Dörnyei, Z. 2005. The Psychology of the Language Learner: Individual Differences in Second Language Acquisition. Mahwah, NJ: Lawrence Erlbaum. Kormos, J. and A. Trebits. 2012. ‘The role of task complexity, modality and aptitude in narrative task performance’. Language Learning 62/2: 439 – 472. Weissberg, R. 2008. ‘Critiquing the Vygotskian approach to L 2 literacy’ in D. Belcher & A. Hirvela, A. (eds. ). The Oralliterate Connection: Perspectives on L 2 Speaking, Writing, and Other Media Interactions. Ann Arbor: University of Michigan Press. Williams, J. 2008. ‘The speaking-writing connection in second language and academic literacy development’ in D. Belcher & A. Hirvela, A. (eds. ). The Oral-literate Connection: Perspectives on L 2 Speaking, Writing, and Other Media Interactions. Ann Arbor: University of Michigan Press.