Discussing study limitations in Applied Linguistics research A

Discussing study limitations in Applied Linguistics research: A corpus-based comparison of Masters dissertations and published research articles Philip Nathan English Language Centre Durham University BAAL Corpus SIG Durham 2017

Limitations Components in Research Writing � Limitation components identified in RAs in a range of disciplines. � Forming elements of Discussion/Conclusion sections. � Limited research on these limitations components beyond identification.

Prevalence/Frequency of Occurrence � Bunton, (2005) examined 45 doctoral theses conclusions sections � Limitations moves were identified in more than 25% of conclusions in ‘thesis-oriented science and technology theses and theses in social sciences and also in field-oriented theses. � Three explicitly titled sections labelled ‘limitations’ were identified.

Prevalence/Frequency of Occurrence � Dudley-Evans (1994) identified in the context of masters dissertations in the biological sciences, the presence of 3 broad moves in discussions sections, namely Introduction, Evaluation of results and Conclusions and future work. [But according to Peacock (2002) – empirical data is not presented to support his structure] � The evaluation of results move comprised in itself 9 moves one of which was identified as a limitations move. � Moves occurred in a cyclical pattern within the evaluation move

Prevalence/Frequency � � � Yang and Allison (1993) identified a limitations move in 1 results section within an evaluating the study move. They identify stating limitations as part of an ‘evaluating the study’ move within RA discussion sections in applied linguistics research articles. This occurred in 25% of the 8 discussion sections they looked at. In conclusions sections, 7 of 13 conclusion sections contained limitations elements within an Evaluating the study move but also separate sections called pedagogical implications within which two limitations moves (referred to as steps) were identified within the Evaluating the study move. Overall 40% of the texts examined in this research contained limitations components

Prevalence/Frequency � Limitations example, � Hill. components also identified in for Sopsela and West (1982) in experimental RAs � Tessuto (2014) – in 40% of law RAs � Kanoksalipatham (2005) – in 80% of biochemistry RAs � Peacock (2002) in 43% of RA Conclusion sections across 7 disciplines 9 - from 31. 5% in Environmental Science RAs to 62. 5% in Language and Linguistics/

Prealence/Frequency � Basturkmen (2012) identifies no limitations sections in Dentistry RAs � Stoller and Robinson (2013) identify no limitations components in Chemistry RAs

What are limitations? �A limitation is an element usually tied to research methodology which limits or reduces the level of generalisation that can be applied to research conducted � which and/or represents an element of the research which provides a potential threat to the validity of the findings and conclusions.

Purpose of limitations components: � Thompson (2005) identifies limitations as one element of a conclusions move titled ‘consolidating the research’ with one purpose of the conclusions section identified as being ‘to forestall criticisms by stating the limitations of the research’.

Rhetorical positioning � Limitations is either placed within an ‘Evaluation of results/research’ move found in the Discussion/Conclusion section of RAs. (Dudley. Evans, 1994) � This move is found in a 3 -move structure: Introduction, Evaluation of results and Conclusions and future work � Swales moves and Feak (1994) suggest 3 conclusions Consolidate the research space, limitations, further research

Within the limitations move � Bitchener (2010: 207) uses the following 3 moves within the limitations component: limitation – significance – further research

Motivation for the Study � On our MA TESOL/Applied Linguistics for TESOL, limitations sections written by MA dissertation seemed rather weak. � Pedagogical aim of improving the writing of these components by these MA students. � A poor limitations section can fatally undermine research writing. � While recognising different contexts and purposes, set out to compare published limitations components with MA limitations components.

Sample of a limitations text which could be (significantly) improved � Primarily, the sample size is not large enough to generalize the results. It may not reflect the different situations regarding to different teachers’ perception, practice and students’ opinion and preference that across diverse contexts. Thus, it can be only [used] as a reference of WCF studies in Chinese middle school context. In addition, the sample size of the interview is too small, which means some results present huge individual differences and [are] difficult to generalize… Last but not least, there are some design flaws in the questionnaire design. Even though we made an effort to distinguish ‘opinions’ and ‘facts’ and use the percentages to present all the statistics. [sample 0061415]

Sample of a limitations text which could be (significantly) improved � Firstly, in this study, there is a small-size population, which may make the result inaccurate because the data is not enough: only 45 participants [were] involved this study. …Secondly, another limitation of this study is the short span of the speaking course. Since the study is conducted in a speaking class in university, we [didn’t] have the additional time and the peer feedback sessions are limited. [sample 0151415]

Corpora � 18 [30] MA Applied Linguistics Limitation Components [6416 words] approx. 90% NNS � 30 RA Limitations Components – balanced from ESP J, JEAP, JSW [ 6802 words] Dissertation average word count = 356 Journal average word count = 227

Communicative purposes (following Askehave and Swales, 2001) � Initial hypotheses: RAs - “to forestall criticisms of the research supporting research validity such that the RA will be considered acceptable by reviewers and readers and therefore be published” Dissertations - “to demonstrate knowledge and understanding of research principles to examiners and to forestall criticisms of the research such that examiners consider the work to be valid”

Hypotheses – Explanations for perceived weakness � Significantly different rhetorical moves and move frequencies would occur in RAs and MA dissertations � MA dissertations would have lots of negative language compared to published RAs. � MA dissertations would contain long lists of limitations compared to published RAs.

Signalling of Limitations � Limitations and pedagogical applications � Limitations and future research � Limitations of the study In RAs limitations most frequently appeared as elements of conclusion sections

Limitations Sections: Overall Move Structure(s) � See handout

Prototypical RA Limitations Structure � See handout [text to move]

Move count comparison (selected moves) Moves RA frequency (26) Diss Freq (16) Signalling consideration of limitations 52% 67% Strengths/achievements (all) 56% 73% Rationale 48% 87% Impact 48% 53% Remedy (all) 72% 80%

Consideration � Dissertations contained many of the different moves more frequently than in RAs. � Longer texts (15000 v 7000 for journals) so there’s more to write and this might suggest a greater variety of moves are needed.

Frequency of Limitation Cycles � Average 2. 12 cycles per sample in RAs � Average 3 cycles per sample in dissertations � Not necessarily of significance in terms of rhetorical move patterns, as dissertations are longer texts.

Keyword analysis � can/ may/could frequencies, might indicate more hedging in dissertations.

But also notably: Lemma RA Diss only 0. 29 0. 46 too 0. 03 0. 08 although 0. 13 0. 08 though 0. 06 0. 02 despite 0. 10 0. 03 even though 0. 01 0. 00 nevertheless 0. 06 0. 00 however 0. 29 0. 45 but 0. 09 0. 23 0. 19 0. 06 Limiters Concessives/ Contrast markers Lexis Explore

Use of ‘Relatively’ � The findings of this study should be interpreted with caution, not only because of the relatively small size of the study but also because of the nature of the writing tasks used. [RA LIM 2] � The limitation of the study may result from the relatively small database of only 25 medical RAs, …[RA Lim 8] � Nevertheless, there are some limitations associated with the study, specifically the fact that the validating corpora are relatively small. [RA Lim 4]

Expressing strengths: RAs � Although the larger national study provided a wealth of rich data across a broad spectrum of students in the U. S. , … [ RA LIM 1] � Finally, even though this study makes valuable contributions to advance L 2/FL writing research, … � [RAJSW 1] � Although the present study has yielded a number of significant implications concerning a link between writing-to-learn and learning-to-write, … [RAJSW 8] � Despite the inherent limitation of a single case, however, we believe that the description of the teaching context of the teacher’s genre work implementation and the detailed SFL analyses of Ji Soo’s texts before and after the genre work provide rich evidence for…… [RAJSW 9]

Expressing strengths: Dissertations � Therefore, the results and conclusion derived from this triangulation method are more valid. [sample 0041314 ] � This research has achieved the purpose and answered the research questions, however… [sample 0081314] � By employing semi-structured interviews, 8 Chinese students’ perspectives on peer feedback were exposed to this study, which largely satisfies research questions. Simultaneously, some new findings were also revealed through interviewing target learners. However. . [sample 013]

‘too’ – in dissertations � In addition, the sample size of the interview is too small, which makes some results present huge individual differences and difficult to generalize. [sample 0061415] � In fact, some respondents commented that the questionnaire was too long for them to finish patiently. [sample 011]

‘Only’ in dissertations � only 45 participants [were] involved this study � as the ones who could respond the interview were only two , due to the lack of free time. [sample 0151415] � However, I only managed to collect data from three universities, [sample 011] � All feel rather negative!

‘Only’ in RAs: Positive spin? The limitation of the study may result from the relatively small database of only 25 medical RAs, … [ESPJ LIM 8] [It’s not small, it’s ‘relatively small’] � Given the labour intensive nature of the analysis only a limited number of writing samples could be examined in this exploratory study. [JEAP LIM 4] [this is an exploratory study so the number of writing samples doesn’t matter so much/and we did a lot of hard work!] � One of them is that we conducted only one interview with some of the participants. [JEAP LIM 1] [but we conducted more than one interview with most of them] �

Conclusions 1. MA dissertations would have lots of negative language compared to published RAs. - some greater use of ‘only’ ‘too’ but perhaps more prevalent is lack of mitigation/spin to reduce the impact of negative statements and reduce the sense of negativity. 2. MA dissertations would contain long lists of limitations compared to published RAs. There are on average more limitations mentioned in MA dissertation texts, but this is considered due to these texts being longer and more needing to be written.

Conclusions � No significant differences in frequency of deployment of rhetorical moves were identified in this work. � There appear to be differences in the use of concessives between MA and RA writers. � Professional/published RA writers have a range of strategies for mitigating limitations and for making limitations/problems more palatable. � There’s evidence that students understand the formal rhetorical moves required, but they need linguistic support in realising moves and spinning their statements.

Pedagogical Implications � Teach/familiarise students with rhetorical structure/elements of limitations sections. � Provide full, complete and accurate communicative purposes of these limitations elements. � Teach students tactics strategies for mitigating their limitations. � Teach students about the realities of research writing (persuasion, selectivity, integrity etc)

Future work � Refining of the rhetorical move model � Co-rater analysis regarding (a) the rhetorical move structure (b) quality of limitation samples � Interviews with student dissertation writers � Interviews with RA writers � Fine-grained linguistic analysis of both RA and Dissertation samples to identify rhetorical tactics and mis-steps � Analyse doctoral limitations texts

References � � � Dudley-Evans, T. (1994). Genre analysis: An approach to text analysis for ESP. In M. Coulthard (Ed. ), Advances in written text analysis (pp. 219– 228). London: Routledge. Hewings, M. (1993). The end! How to conclude a dissertation. In G. M. Blue (Ed. ), Language, learning and success: Studying through English. London: Modern English Publications and The British Council, Macmillan. Hill, S. S. , Soppelsa, B. F. , & West, G. K. (1982). Teaching ESL students to read and write experimental research papers. TESOL Quarterly, 16(3), 333– 347 Holmes, R. (1997) Genre analysis and the social sciences: An investigation of research article discussion sections in three disciplines English for Specific Purposes 16 (4) 321 -337 Peacock (2002) Communicative moves in the discussion sections of research articles System 30 479 -497

References � � Stoller and Robinson (2013) Chemistry journal articles; An interdisciplinary approach to move analysis with pedagogical aims English for Specific Purposes 32 (2013) 45 -57 Tessuto, G. (2014) Generic structure and rhetorical moves in English language empirical law research articles: Sites of interdisciplinary and interdiscursive crossover. Thompson, P. (2005) Points of focus and position: intertextual reference in Ph. D theses’ Journal of English for Academic Purposes 4: 3017 -323 Yang, R. , & Allison, D. (2003). Research articles in applied linguistics: Moving from results to conclusions. English for Specific Purposes, 22, 365– 385.