Another peek into what todays English learners as

Another peek into what today’s English learners as researchers actually do in production with the scaffolding of 3 e-referencing tools Hsien-Chin Liou and Chih-Hsin Lee hcliu@mx. nthu. edu. tw CALICO 2006 Foreign Languages and Literature, National Tsing Hua University 1

Overview n Purpose of the study n Literature Review n Research Questions n Data Collection n Data Analysis n Conclusion 2

Purpose of the Study n Recently the pedagogical effectiveness of referencing tools in assisting language learning has attracted quite a lot of attention (e. g. , Rundell, 1999; Yang, 2005). n The aim of this study is to investigate the outcome and process of 22 college learners while doing writing and translation tasks. Their preferences and look-up strategies of using 3 web-based referencing tools were documented. The tools were a bilingual dictionary, a bilingual concordancer, and a collocation retrieval program. 3

Literature Review- corpus investigation n Chambers (2005) examined the data of the students' consultation of the corpora, including choice of search word(s), analytical skills, the problems encountered, and their evaluation of the activity. n Although corpora consultation can complement foreign language learning in various educational contexts, limitations were also found such as the small size of corpora and lack of learner training. 4

Literature Review. Dictionary Look-up Behavior n Dictionary look-ups have been demonstrated as useful scaffolds when learners are working on language tasks (Laufer & Hadar, 1997; Rundell, 1999). n Factors of word relevance, word inferability and learners’ vocabulary knowledge were shown to influence look-up behavior (Hulstijn, 1993). n Inferring ability is related to word consultation in a less straightforward manner than is learners’ existing vocabulary knowledge. 5

Literature Review. Dictionary Look-up Behavior n Rundell (1999) recommended learners to use monolingual dictionaries rather than bilingual dictionaries due to their richer information about syntactic behavior, collocation, word frequency, or synonyms. n The bilingualised dictionary yielded the best results on both comprehension and production (Laufer & Hadar, 1997). n When foreign-language readers look up the meaning of unfamiliar words can be influenced by the language task and learner variables. 6

Literature Review. Dictionary Look-up Behavior n Frankenerg-Garcia (2005) examined how learners used multiple types of reference materials altogether while doing translation. He used a detailed grid for types of query on various tools and found bilingual types were preferred. n Along the same line of Bland, Noblitt, Armington and Gay (1990), Hulstijn (1993), and Liou (2000 a), Liou (2000 b) claims that the use of online recording programs to do unobtrusive observation of learner behavior can yield valuable data for applied linguists in understanding what learners actually do on tasks. 7

Literature Review- corpus investigation n As technologies advance, online corpus consultation has become popular and proved useful for language learning. n Previous studies have shown that the concordancer is a useful tool to enhance L 2 learners’ vocabulary learning (Cobb, Greaves, & Horst, n. d. ; Kennedy & Miceli, 2001; Lee & Liou, 2003) and writing (Gaskell & Cobb, 2004; Kennedy & Miceli, 200; Yeh & Yu, 2004; Yoon & Hirvela, 2004). 8

Literature Review- corpus investigation n More research is required to investigate how learners behave while working on different tasks with the help of various e-reference tools. n The effectiveness and students’ preferences of using different on-line reference tools should be addressed with mixed research methods so that a more comprehensive picture can be revealed. 9

Research Questions 1. Does the online writing practice have an impact on the students’ performance? 2. What can the computer recording data tell us about the students’ look-ups as a function of tool and task differences? 3. What are the students’ preferences and perceptions of using three online referencing tools in order to complete translation and writing tasks? What are the benefits and difficulties of using them? 4. What does the process data based on the analyses of the selected students’ think aloud protocols tell us? 10

The Study: (1) Participants n Twenty-two first-year English majors from an intact class of a public university participated in this study. n Most of them did not have previous experiences in CALL or corpus learning. n Five voluntary subjects out of the 22 subjects joined two think-aloud sections. 11

(2) Tools and Tasks: 3 Web-based Referencing Tools TOTALrecall TANGO Dr. eye Features A bilingual concordancer A collocation retrieval program A bilingual dictionary Output Bilingual text Monolingual definition with bilingual examples Corpus Sinorama (1990 -2000), Voice of America, and British National Corpus (see writing practice on Candle for more detailed information http: //candle. cs. nthu. edu. tw ) 12

Bilingual Concordancer: TOTALrecall 13

Bilingual Collocation aid program: TANGO 14

A commercial bilingual dictionary, popular in Taiwan: Dr. eye 15

Figure 1: Illustration of the online writing practice unit 16

Figure 2: Illustration of the online translation unit 17

Figure 3: Illustration of the online writing unit 18

(3) Data Collection Quantitative Data • the scores of students’ pre-writing and postwriting • records of the tracker on the frequency of look-up entries • evaluation questionnaires Qualitative Data • 2 sections of thinkaloud protocols of five participants 19

(4) Procedure of the Study Date Procedures 9/14 Pre-writing 9/21 Orientation of TANGO & TOTALrecall 9/28 Translation (1) 10/12 Writing (1) 10/14 Think-aloud training 10/19 Translation (2), Think-aloud (1), & Evaluation questionnaire (1) 10/26 Writing (2), Think-aloud (2) & Evaluation questionnaire (2) 10/28 Post-writing 20

n (5) Data Analyses--Quantitative data analysis (5. 1) Scoresprewriting of Prewriting and Post-writing Students’ and postwriting were graded by two experienced raters using the scale of grading the Test of Written English by the Educational Testing Service with the criteria of 1 to 6. Table 2. Comparison of pretest and post writing Pre-test Post-test Mean 3. 19 4. 21 SD 0. 68 0. 61 Inter-rater Correlation 0. 83 0. 92 21

Quantitative Data Analysis (5. 2) Scores of Prewriting and Post-writing n Due to the small number of test takers, we used the nonparametric procedure-Sign Test. Table 3. Comparison of writing performance at 2 time points Sign Test Marked tests are significant at p <. 05000 Pretest & Posttest No. of Percent Z p-level 22 4. 050814 0. 000051* 95. 45455 22

Quantitative Data Analysis (5. 3) Tracking Data—Language Use Table 4. The summary of language used for queries for different tasks Tasks Chinese Writing 622 (55. 9%) 27 461 (41. 4%) 3 1113 Translation 1109 (66. 5%) 30 520 (31. 2%) 8 1667 1731 57 981 11 2780 Sum Both English Error Sum It seems that the learners used tools more frequently in translation tasks. The number of tool use in translation is about 1. 5 times of it in writing tasks. 23

Quantitative Data Analysis (5. 4) Tracking Data- Language Use n Users looked up more entries in Chinese for translation, but more English look-ups appeared in writing. This phenomenon may be related to the nature of the tasks. n In translation tasks, users needed to translate the Chinese text into English so that it was necessary for them to use Chinese to search for the English equivalents. n In writing tasks, the learners needed to confirm what they want to express in English so that the frequency of English mode in writing is higher. 24

Quantitative Data Analysis (5. 5) Tracking Data- Tool Use Table 5. The use of different tools in different tasks Type Dictionary (Dr. eye) Collocati Concorda on aid ncer (Tang (Totalre o) call) Sum Writing 50. 85% 2. 16% 46. 99% 100% Translation 51. 53% 4. 98% 43. 49% 100% n The frequency of use of Dr. eye (the dictionary) and TOTALrecall (bilingual Sum 51. 26% 3. 85% 44. 89% 100% concordancer) were considerably higher in both writing and translation tasks. . 25

Quantitative Data Analysis (5. 6) Tracking Data- Tool Use n Seven out of the 22 learners preferred the use of the bilingual concordancer (TOTALrecall) than the other two tools, while ten of them showed the preference for the dictionary (Dr. eye), and the rest of them used the dictionary (Dr. eye) and TOTALrecall in similar frequency. n Possible factors that influence individual difference in terms of such preference may include learners’ familiarity towards different tools, the design of the tools and the search results, or usefulness of searched information presented by different tools. 26

Quantitative Data Analysis (5. 7) Tracking Data n The tracker also recorded students’ use of the online referencing tools within one month after the post -writing test. n After the experiment, 12 out of 22 students continued to use these tools. Table 6. Students’ use of the tools one month after the experiment Total Percentage TOTALrecall 123 70. 69% TANGO 38 21. 84% Dr. eye 13 7. 47% Sum 174 100% 27

Quantitative Data Analysis (5. 8) Tracking Data Table 7. The summary of language used for queries one month after the experiment English Total Percentage n 86 49. 43% Both 1 0. 57% Chinese 83 47. 70% Error 4 2. 30% Sum 174 The languages of students used one month after the experiment while querying were roughly in the same frequency. 28

Quantitative Data Analysis (5. 9) Evaluation Questionnaire – Usefulness of Tools Table 8. 22 Participants’ perceptions of usefulness of the 3 tools or their combinations TANGO TOTALrecall (1) (2) Writing 0% 13. 63% Translation 4. 55% n Dr. eye (3) 1+2 1+3 2+3 1+2+3 22. 72% 9. 09% 4. 55% 36. 36% 13. 63% 4. 55% 18. 18% 40. 90% 22. 72% Most participants thought that only one single e-referencing tool might not be enough to assist them and using more than one tool is the most useful strategy. 29

Quantitative Data Analysis (5. 10) Evaluation Questionnaire. Usefulness of Tools n Dr. eye is more helpful in learning the meaning of the vocabulary. n TOTALrecall is helpful in learning the usage of vocabulary. TANGO is the second helpful tool, and Dr. eye is the least. n TANGO, for the participants, is a more helpful tool in learning collocation due to its collocation patterns. 30

Quantitative Data Analysis (5. 11) Evaluation Questionnaire. Usefulness of Tools n The participants think that using e-referencing tools is more helpful in improving Chinese. English translation skill than English writing skills. n Dr. eye is the most helpful tool in improvement of translation, TOTALrecall the second. 31

Quantitative Data Analysis (5. 12) Evaluation Questionnaire. Usefulness of Tools n Participants think that TANGO is the most helpful tool in improvement of English writing skill (which contradicts to what they would prefer to use in the future). n The participants reported that they did not know how to use TANGO because they were not very familiar with the functions of the tool. n More intensive orientation and practice would be the first priority for students to familiarize the functions of the tool. 32

Quantitative Data Analysis (5. 12) Evaluation Questionnaire n For future use, students generally hold positive attitudes towards using these e-referencing tools in the future. n Mostly they would like to have more than one e -referencing tools, especially when doing translation tasks. 33

Qualitative Data Analysis (1) Think-aloud Protocols n A coding scheme adopted from Frankenberg-Garcia (2005) and modified was taken up in order to identify and classify their look-up strategies. The coding scheme includes seven categories: 1. Finding an L 2 equivalent; 2. Confirming a hunch; 3. Finding a suitable collocate; 4. Choosing the best alternative or synonym; 5. Checking spelling; 6. Retrieving linguistic info from examples; 7. Looking for part of speech. n 34

Qualitative Data Analysis (2) think-aloud protocols n Each individual concept was identified, in terms of each individual problem to be solved in writing or translation no matter how many entries they entered. n Regardless of task difference in writing or translation, students’ look-up behavior falls into three major categories: finding an L 2 equivalent (50%), confirming a hunch (19%) and retrieving linguistic information from examples (11%). 35

Qualitative Data Analysis (3) think-aloud protocols n In order to obtain information students needed, they tended to do cross referencing very often, which is also reflected in our tracker data. n Each e-referencing tool provides different types of linguistic information. The information each ereferencing tool provided might not be sufficient. Cross-referencing could help students to test and confirm their hypothesis while searching. 36

Qualitative Data Analysis (4) Think-aloud Protocols n Successful use of strategy was defined as finding the correct information and applying it in writing or translation correctly. n The results show that 89% of these 5 students’ problems were solved and applied successfully. 37

Qualitative Data Analysis (5) think-aloud protocols n Among 89% of successful use of strategy, 3 categories: finding an L 2 equivalent (43%), confirming a hunch (18%), and checking spelling (9%) were found to be more often used. n Other strategies that were not used successfully might need to be trained. 38

Triangulation of All the Data (1) n The participants also reported that when having problems in English writing, they would tend to search Dr. eye first, then TOTALreacll. n And Dr. eye and TOTALrecall could usually provide the information they needed. n The data confirmed that they would do initial look-up of perhaps an English equivalent of a Chinese entry first, followed by further consultation of how the word should be used in context using either TOTALrecall or TANGO. 39

Triangulation of All the Data (2) n We observed several different types of problems students had while searching for information: unfamiliarity with tools themselves, or with searching techniques, the small size of corpora in the tools, over-reliance on the tools, and carelessness in searching. n The questionnaire responses also pointed to some difficulties that students encountered while using these ereferencing tools. Only 22. 72% of students perceived no difficulty in using the tools, whereas 36. 36% of students felt that they had difficulty in using TANGO and 18. 18% of students in using TOTALrecall. n 40

Conclusion n The statistical comparison indicated this group made significant improvement (the learning products) perhaps through the design and the use of the online writing practice tasks. n It seems that online writing practice with scaffolds of referencing tools can improve their production as evidenced in the learning product data. 41

Conclusion n The acceptance of the innovative corpus tools for references is satisfactory to the learners based on the questionnaire data; definitely more orientation with practice from English teachers is needed for students to acquire mature look-up skills. n Although succinct task differences between writing and translation were not observed, they warrant more future research. 42

Acknowledgements n The paper was funded by a National Science Council project (under the number of NSC 94 -2524 -S 007001, the CANDLE project). n We also thank Hsin-Ping Yu for programming and sorting the tracked data, Jany Lin for helping collecting and analyzing the data, and Ashley Ho for serving as another rater for grading the writing and coding the think-aloud protocols. n The 22 participating students are also acknowledged. 43

Thank You for Your Attention! 44