FORTY YEARS OF RESEARCH ON INSTRUCTED LANGUAGE LEARNING

FORTY YEARS OF RESEARCH ON INSTRUCTED LANGUAGE LEARNING: WHAT HAVE WE LEARNED SO FAR? LUKE PLONS KY NORTHER N AR IZONA UNIVERSITY CILLT, 2018 UNI VERSIDA D NACIO NAL A UTÓNOMA DE MÉXICO

Modeling L 2 development Learner INTERNAL mechanisms/processes Learner EXTERNAL mechanisms/processes ? explicit instruction implicit SA immersion context at-home class Instruct. context lab +/-amount input type working memory SL environment FL ? emotions beliefs about language learning ATI anxiety L 2 strategies tech medium F 2 F stress personality relative clauses Language complexity aptitude FSs intelligence motivation 2

(Maybe the most obvious thing you’ll hear all week) L 2 research does not always benefit L 2 practice. Why? At least 9 reasons

L 2 research does not always benefit L 2 practice. Why? 1. There are many competing models. 2. We’re a young field. 3. Not all research is concerned with practice (and that’s OK!) 4. Some researchers are unable to find relevance. 5. Science self-corrects over time (or at least it should), so our understanding of L 2 teaching and learning should too. 6. Some research doesn’t align with our educational culture or local/curricular constraints. 7. Some research is biased, flawed, or fraudulent. 8. Many practitioners do not read current research. 9. No single study can take into full consideration the multivariate nature of language … or the massive amounts of findings accumulating

L 2 research does not always benefit L 2 practice. 1. There are many competing models UG Usage-based learning Skill-acquisition theory Competition model Input processing Interactionist model Monitor model Sociocultural theory Processability theory Complexity theory

L 2 research does not always benefit L 2 practice. 2. We’re a (theoretically) young field Before some applied questions can be answered, certain basic, theoretical questions need to be addressed regarding the nature of language learning, teaching, use, assessment, etc. Some (many) potentially useful questions have not yet been addressed, or not sufficiently.

L 2 research does not always benefit L 2 practice. 3. Not all research is concerned with practice (and that’s OK!) (Personal opinion: AL is at its best when concerned with real-world issues but…) Research for the sake of knowledge construction has a place. (We don’t question the value of theoretical physics, right? ) And forcing undue relevance is actually unethical.

L 2 research does not always benefit L 2 practice. 4. Some researchers are unable to find relevance

L 2 research does not always benefit L 2 practice. 5. Science self-corrects (so our understanding of L 2 teaching and learning should, too. ) Again, premature application might do more harm than good! (More on the need for replication as a means to self-correction later!)

L 2 research does not always benefit L 2 practice. 6. Some research doesn’t align with our educational culture or local/curricular constraints the study your context

L 2 research does not always benefit L 2 practice. 7. Some research is biased, flawed, or fraudulent Researchers are human Some researchers are also unethical Plonsky (2017/in prep) - Survey of 300+ authors of published AL research 17% admitted to falsifying or fabricating data - _____ (1. 7% in other fields; Fanelli, 2009)

L 2 research does not always benefit L 2 practice. 8. It’s hard impossible to keep up with the literature. I’m not blaming anyone! I can’t keep up either. 40 journals x 4 issues/year x 5 articles per issue = 800 articles/year x 2 hours/article = conservatively, 1600 hours per year?

L 2 research does not always benefit L 2 practice. 8. It’s hard impossible to keep up with the literature. Marsden & Kasprowicz (2017); N = 391: What prevents you from reading? Lack of time Lack of digital access Lack of conceptual access Enter

L 2 research does not always benefit L 2 practice. 8. It’s hard impossible to keep up with the literature. Marsden & Kasprowicz (2017); N = 391: What prevents you from reading? Lack of time …. . . . . SHORT (1 page max) Lack of digital access …. . . . FREELY ACCESSIBLE Lack of conceptual access …. . . . . WRITTEN IN NON-TECHICAL ENGLISH oasis-database. org

L 2 research does not always benefit L 2 practice. 9. No single study can take into full consideration the multivariate nature of language … or the massive amounts of findings accumulating … Three solutions 1. Statistical Use of multivariate procedures for analyzing (quantitative) data. (Qualitative researchers are generally quite good at examining relationships holistically. ) learning, teaching, use, processing, assessment, policy, curriculum, etc. 2. Methodological An empirical approach for synthesizing and making sense of results across bodies of research and the interactions between variables they examine. 3. Cultural/conceptual A mindset for considering and integrating empirical findings. Metaanalysis Syntheticmindedness

Modeling L 2 development Learner INTERNAL mechanisms/processes Learner EXTERNAL mechanisms/processes ? What has meta-analysis told us? (Spoiler alert: Instruction works! But how much and when and under what SL explicit environment circumstances? ) instruction FL implicit SA immersion context at-home class Instruct. context lab +/-amount input type working memory ? emotions beliefs about language learning ATI anxiety L 2 strategies tech medium F 2 F stress personality relative clauses Language complexity aptitude FSs intelligence motivation 16

Wait! What is metaanalysis?

Assumption: Developing scientific knowledge is a cumulative and corporate enterprise. What is meta-analysis? Empirical approach to reviewing literature More systematic and objective than traditional reviews. Origin? (“necessity is the mother of invention”) Study 1 Study 2 Study 3 Meta-analytic result!

What is meta-analysis? Assumption: Developing scientific knowledge is a cumulative and corporate enterprise. THREE hallmarks (Plonsky & Oswald, 2015) 1. Exhaustive (vs. selective) searches (sample ≈ population) ◦ Validity generalizability 2. Systematic coding for substantive features and effects (vs. subjectively or idiosyncratically interpreted) 3. Key component: effect sizes (e. g. , d, r) more precise, stable, intuitive, and informative (vs. p) Consequently…

Meta-analyses provide stable , trustworthy, and precise answers! Q: Is computer-based feedback helpful? A: Yes! Just as helpful or more so than face-to-face feedback: d = 1. 13 vs. d =. 84 (Ziegler, 2013; K = 14). Q: Is strategy instruction effective? A: YES, very effective, especially for metacognitive strategies and for reading and speaking; less so for listening or writing (Plonsky, 2011, in press)

Meta-analyses provide stable , trustworthy, and precise answers! Q: Is it helpful to provide students with feedback when they make errors in class? A: YES, but it depends on what type of feedback you provide (Lyster & Saito, 2008)

All: Cooper & Hedges (2009) Medicine: Ioannidis (2016) Growth of MA across the sciences… Ecology: Stewart (2009) Economics: Poot (2012)

…in the broader language sciences, … 250 232 200 LLBA 150 119 100 50 228 30 56 38 25 0 1980 - 1985 - 1990 - 1995 - 2000 - 2005 - 2010 -

And in applied linguistics (see lukeplonsky. wordpress. com/bibliographies/meta-analysis) 150 125 100 92 50 0 19 2000 -2005 2006 -2011 2012 -In Press

OK, so once again… What do we know?

Spada & Tomita (2010 d; k=9) Spada & Tomita (2010 c; k=29) 0. 33 Goo et al. (2015 c; k=27) 0. 60 Spada & Tomita (2010 b; k=20) 0. 73 Spada & Tomita (2010 a; k=24) 0. 88 Norris & Ortega (2000; K=45) Grammar Good news: L 2 instruction can be very effective! 0. 39 0. 96 Shintani (2015 c; k=23) 1. 01 Goo et al. (2015 a; k=27) 1. 06 Shintani et al. (2013 b; k=21) 1. 13 Shintani et al. (2013 c; k=22) 1. 23 Shintani et al. (2013 d; k=22) 1. 32 Shintani et al. (2013 a; k=20) 1. 96 Shintani (2015 b; k=37) 2. 03 Shintani (2015 d; k=23) 2. 52 Vocab Shintani (2015 a; k=42) 2. 60 Won (2008; K=30) 0. 69 Chiu (2013; K=16) Jeon & Kaya (2006; K=13) Prag. Let’s take a closer look 0. 75 0. 59 Plonsky & Zhang (in press; k=50) 1. 68 Goo et al. (2015 d; k=4) 1. 86 Goo et al. (2015 b; k=4) Pron. Sakai & Moorman (2017; prod) 2. 74 0. 54 Lee et al. (2015; K=79) 0. 82 Sakai & Moorman (2017; perc) 0. 00 0. 92 0. 50 1. 00 1. 50 2. 00 2. 50 3. 00

1. L 2 grammar instruction

Grammar instruction Overall effects d =. 96 (Norris & Ortega, 2000; k = 49) d = 1. 095 (Goo et al. , 2015; k = 34) OK, but what else?

Grammar instruction Moderator analysis: Different types of instruction EXPLICIT > IMPLICIT Norris & Ortega (2000)

Grammar instruction Moderator analysis: Different types of instruction EXPLICIT > IMPLICIT Goo et al. (2015)

Grammar instruction Moderator analysis: Different types of instruction and target features EXPLICIT > IMPLICIT Spada & Tomita (2010)

Grammar instruction Moderator analysis: Different types of outcome measures Controlled > Spontaneous Norris & Ortega (2000)

Grammar instruction Moderator analysis: Different types of outcome measures Controlled > Spontaneous Implications for assessment? Goo et al. (2015)

2. L 2 vocab instruction

Vocab instruction Overall effects d =. 69 (Won, 2008; k = 30) d =. 75 (Chiu, 2013; k = 16) OK, what else?

Vocab instruction Moderator analysis: different types of instruction L 1 support > no L 1 support Implications for materials development? Won (2008)

Vocab instruction Moderator analysis: different types of learners HS/Uni students > Elementary students Implications for curriculum, policy? Chiu (2013)

3. L 2 pragmatics instruction

Pragmatics instruction Overall effects d = 0. 59 (Jeon & Kaya, 2006; k = 13) d = 1. 52 (Plonsky & Zhuang, in press; k = 50) What else?

Pragmatics instruction Moderator analysis: Different types of instruction EXPLICIT > IMPLICIT 1. 91 vs. 1. 01 (Jeon & Kaya, 2006) 1. 68 vs. 1. 27 (Plonsky & Zhuang, in press) LONGER = BETTER 0. 82 (“long”) vs. 0. 42 (“short”) (Jeon & Kaya, 2006) r =. 54 for length (minutes) x treatment effect (Plonsky & Zhuang, in press)

Pragmatics instruction Moderator analysis: Different types of outcome measures Plonsky & Zhang (in press)

4. L 2 pronunciation instruction

Pronunciation instruction Overall effects Production training (Lee, Jang, & Plonsky, 2015; k = 86) - d =. 69 Perception training (Sakai & Moorman, 2017; k = 18) - Perception: d =. 92 - Production: d =. 54 What else?

Pronunciation instruction Moderator analysis: Different types of instruction Lee et al. (2015) familiar unfamiliar Implications for researchers?

Pronunciation instruction Moderator analysis: Different types of outcome measures Controlled > Spontaneous Lee et al. (2015) Implications for pronunciation assessment?

5. Technology enhanced L 2 instruction

Tech-enhanced L 2 instruction (CMC) (see also Plonsky & Ziegler, 2016) Overall effects Lin (2015)

Tech-enhanced L 2 instruction (CMC) Moderator analysis: Target skill Lin (2015)

Modeling L 2 development Learner INTERNAL mechanisms/processes Learner EXTERNAL mechanisms/processes What about these factors? ? explicit instruction implicit SA immersion context at-home class Instruct. context lab +/-amount input type working memory SL environment FL emotions beliefs about language learning ATI anxiety L 2 strategies tech medium F 2 F stress personality aptitude How would you rank them in terms of relative intelligence their associations with L 2 achievement? clauses Language motivation complexity FSs 49

First, how do we estimate the strength of the association between these constructs and L 2 proficiency? Typical design: 1. Measure (a)—proficiency/achievement 2. Measure (b)—the ID in question 3. Run a correlation between (a) and (b) 1. Positive, negative 2. Weak, moderate, strong

2. How do we estimate the strength of the association between these constructs and L 2 proficiency? Serious challenges to this type of research Most IDs are not directly observable ◦ ◦ Questionnaires (and all their issues!) Difficult to validate instruments Measurement error (unreliability) = always Proficiency = very difficult to measure Interpretation: correlation ≠ causation ◦ Chocolate x Nobel prizes ◦ Motivation x proficiency Correlation values (r) assume linear relationships, but that’s not always the case

Individual differences (learner-internal variables associated with L 2 development) q How strongly are these constructs associated with L 2 proficiency/achievement (as shown via meta-analyses)? Probably not Probably, but very little research Yes! Probably not Aptitude 0. 49 Motivation Anxiety Ca Strategies n be ue l f n i ? ced n 0. 37 -[VALUE] 0. 31 WM 0. 26 0 0. 1 0. 2 0. 3 0. 4 0. 5 Meta-analytic correlations (r) 0. 6

L 2 research does not always benefit L 2 practice. 9. No single study can take into full consideration the multivariate nature of language … or the massive amounts of findings accumulating … Three solutions 1. Statistical Use of multivariate procedures for analyzing (quantitative) data. (Qualitative researchers are generally quite good at examining relationships holistically. ) 2. Cultural/conceptual A mindset for considering and integrating empirical findings. learning, teaching, use, processing, assessment, policy, curriculum, etc. Syntheticmindedness 3. Methodological An empirical approach for synthesizing and making sense of results across bodies of research and the interactions between variables they examine. Metaanalysis

Synthetic mindedness and the synthetic movement in AL (Plonsky & Ziegler, 2016; Norris & Ortega, 2006) No single study… “Meta-analysis represents … major changes in the way psychologists view the general research process” (Schmidt, 1992, p. 1172)

Synthetic mindedness and AL (Plonsky & Ziegler, 2016; Norris & Ortega, 2006) In addition to the use of meta-analysis, this movement manifests itself in: Not a means to find fault but, rather, to - refine results & test generalizability - see Marsden et al. (in press) (a) WHAT we choose to study e. g. , Replication research (e. g. , Plonsky, 2012, 2015; Porte, 2012, SLRF wkshop; special issues in JSLW, CALICO; strands in SSLA, LT; colloquium at AAAL 2017 ) (b) How we study it - Sharing and adapting instruments instead of reinventing the wheel - Focusing on practical significance & effect sizes (e. g. , Norris & Ortega, 2006; Plonsky & Oswald, 2014; EIGHT journals now requiring reporting of ES) (c) How we report and interpret our results - i. e. , as individual studies in a larger domain (not as definitive answers)

More suggestions for improving SLA-practice links Search for answers to your questions! If there isn’t any relevant research, do it yourself! --> Formal? Action? Replication! Team up with researchers Team up with colleagues to collect data Help administrators see the value of research-informed decisions Use IRIS to find/adapt materials (iris-database. org) Consider meta-analytic evidence > primary evidence

Conclusion: In this talk I’ve attempted to lay out ONE part of the solution for helping anyone involved in or with an interest in L 2 research (teachers, researchers, policy-makers, curriculum developers, testers, parents) to a. better understand vast amounts of empirical findings, and b. hopefully make more informed links between research and practice. Specifically for RESEARCHERS, I also hope to have instilled in some way the synthetic/meta-analytic (“we’re all in this together!”) mindset.

¡Muchas gracias! Questions? lukeplonsky@gmail. com Slides from today: lukeplonsky. wordpress. com talks conference presentations

other

1. Defining the domain / locating primary studies 2. Developing and implementing a coding scheme 3. (Meta-)Analysis 4. Interpreting meta-analytic results Four major stages - Parallels to primary research? - Each with numerous decisions

Effect sizes A standardized estimate of the magnitude of a relationship or effect (e. g. , d, r) NHST (p) Unreliable; varies as a function of sample size Uninformative; forces continuous results into a dichotomy Arbitrary convention (. 05) Effect sizes Not dependent on sample size Expresses the magnitude of the relationship in question Continuous; can be compared or combined across studies What we really want to know!

Plonsky & Oswald (2014) – d, r Based on a synthesis of effect sizes in 346 primary studies and 91 meta-analyses of L 2 research (N > 604, 000)… Effect Size Small-ish Medium-ish Large-ish 25 th percentile 50 th percentile 75 th percentile d (between) . 40 . 70 1. 00 d (within) . 60 1. 00 1. 40 r . 25 . 40 . 65

The Synthetic Ethic (e. g. , Norris & Ortega, 2006) What does it mean? ? ◦ …for the different stakeholders (e. g. , editors and reviewers? Primary authors? Researchers trainers? ? ◦ For YOU? ! ◦ …for breadth (of theoretical terrain) vs. depth? ◦ …for (primary). . . lit reviews? analyses? data reporting? interpreting? ◦ Do we behave this way? Why (not)?

Interpreting findings (Plonsky & Oswald, 2014) General and field-specific benchmarks (Cohen, 1988; Plonsky & Oswald, 2014) ◦ Just a starting point! Previous/similar meta-analyses in AL (e. g. , Abraham, 2008; Lee et al. , 2015; Mackey & Goo, 2007) Meta-analyses in other fields SD units Setting (Plonsky, 2011) (Taylor et al. , 2006) (e. g. , Li, 2010; Mackey & Goo, 2007) Length/intensity, practicality (Lee & Huang, 2008; Lee et al. , 2015; Lyster & Saito, 2010; Norris & Ortega, 2000) Methodological features (Plonsky, 2011, 2013, 2014; Plonsky & Gass, 2011)