Are banked cloze items sensitive to discourse level

Are banked cloze items sensitive to discourse level constraints? 深圳大学 外国语学院 郭 丽

§ A study into the validity of a new test format in College English Test, Band 4 (CET 4), a national English test for university students in the mainland of China.

1. Introduction 1. 1 Types of cloze * Fixed-ratio vs rational cloze A bird watcher, in the course _①___ a day’s rambling, almost always finds __②___ species of woodpeckers. Sometimes they reveal ___③___ whereabouts by piercing call notes, sometimes __④___ their steady hammering on tree trunks. (Key: of, several, their, by) ……In __①__, on the other hand, while individuals are important, people also try to do everything they can for the other group members. They do this because it is thought that if the __②__ succeeds then each member will succeed. So a person’s life away from work is still more relevant to his or her job is __③___. (Key: Japan, group, America) * Constructed-response vs selected-response cloze (open-ended vs multiple-choice or banked cloze) * Exact word scoring vs acceptable word scoring

1. 2 Banked cloze in CET 4 A new test format in College English Test Band 4 Old test format (before 2007) (Vocabulary and structure) ** We don’t know why so many people in that region like to wear dresses of such _____ colors. A) low B) humble C) mild D) dull (College English Test Band 4, June 2005)

Example of banked cloze Some years ago I was offered a writing assignment that would require three months of travel through Europe. I had been abroad a couple of times, but I could hardly __1__ to know my way around the continent. Moreover, my knowledge of foreign languages was __2__ to a little college French. I hesitated. How could I, unable to speak the language, __3_ unfamiliar with local geography or transportation systems, set up __4__ and do research? It seemed impossible, and with considerable __5__I sat down to write a letter begging off. Halfway through, a thought ran through my mind: you can’t learn if you don’t try. So I accepted the assignment. There were some bad__6__. But by the time I had finished the trip I was an experienced traveler. And ever since, I have never hesitated to head for even the most remote of places, without guides or even __7___ bookings, confident that somehow I will manage. The point is that the new, the different, is almost by definition __8__. But each time you try something, you learn, and as the learning piles up, the world opens to you. I’ve learned to ski at 40, and flown up the Rhine River in a ___9___. And I know I’ll go on doing such things. It’s not because I’m braver or more daring than others. I’m not. But I’ll accept anxiety as another name for challenge and I believe I can ___10___ wonders.

totally scary moments manufacture constantly claim regret limited balloon (College English Test 4, June 2008) ppt 22 reduced interviews advanced news declare accomplish

§ The purpose of banked cloze component in CET 4—— “ to assess students’ ability to understand employ words at discourse level” (National College English Testing Committee, China. 2006: Syllabus for College English Test Band 4. )

2. Review of literature and Necessity for the study Contradictory conclusions concerning the construct validity of cloze ** can measure text-level processing ability (Ramanauskas, 1972; Chihara et al. , 1977; Bachman, 1985; Jonz, 1991; Fotos, 1991; Chavez-Oller et al. , 1994; Storey, 1997; Yamashita, 2003) ** cannot assess comprehension beyond sentence level. (e. g. , Alderson, 1979; 1980; Kibby, 1980; Markham, 1985)

Relevant research (on rational deletion and banked cloze) Yamashita, Junko. 2003 (Processes of taking a gap-filling test: comparison of skilled and less skilled EFL readers. Language Testing 20: 267 -293. ) *12 Japanese EFL students * Gap-filling test -- key content words and cohesive devices were deleted * Thinking aloud * Text-level information 52% sentence level 17%; clause level 10%; extra-textual 4% * A gap-filling test can be used as a test to measure higher order processing ability.

Relevant research (continued) Two studies into banked cloze procedures by Chinese researchers. Gao Xiaoying et al (2008) * Simultaneous introspection and immediate retrospection * 18 Chinese University students * Information sources: clause (47%) ; text level (23%) sentence (22%) ; extra-textual (8%) * The proficient readers -- context-based reading model the less proficient readers -- word-based approaches. Liu (2009) * Similar to the above in research focus and design * 6 second-year students in a Chinese university * Local syntactic (36%) local semantic (29%); sentence (20%); text-level (16%) information.

Relevant research (continued) The present study adopts an approach which share similarities with Jonz (1991) and Chihara et al (1977) —— examining the sensitivity of the cloze items to the disruptions to the normal order of texts. Scrambled-text research / on standard fixed-ratio cloze

Need for this study § Little research has been conducted into banked cloze procedure and consequently there is little evidence as to whether it is sensitive to inter-sentential constraints as the CET 4 designers claim it to be. § The previous research on banked or rational deletion cloze is mostly qualitative, involving only a small number of subjects. § As a high-stake national exam, CET 4 exerts a powerful backwash impact upon the English instruction and testing at tertiary level in China.

Research questions 1) Do subjects use text-level information in completing the banked cloze test? 2) What types of information (within-sentence or beyondsentence) are used to answer each item? 3) Does English proficiency affect the use of text-level information?

3. Study One Materials Normal Test One *Reading in Depth Section, College English Test Band 4 Dec, 2007 *222 words long *10 contents words (4 nouns, 2 verbs, 2 adjectives and 2 adverbs) deleted Scrambled Test One *Produced by scrambling the sentences in Normal Test One *Cause maximum interruption of any constraint on comprehension attributable to sentence sequence. Response and scoring Students were required to select one word for each blank from the list of choices. Since no change in the form was necessary, the scoring was entirely objective.

Normal Test One As war spreads to many corners of the globe, children sadly have been drawn into the center of conflicts. In Afghanistan, Bosnia, and Colombia, however, groups of children have been taking part in peace education __1__. The children, after learning to resolve conflicts, took on the __2__ of peacemakers. The Children’s Movement for Peace in Colombia was even nominated (提名) for the Nobel Peace Prize in 1998. Groups of children __3___ as peacemakers studied human rights and poverty issues in Colombia, eventually forming a group with five other schools in Bogota known as The Schools of Peace. The classroom __4__ opportunities for children to replace angry, violent behaviors with __5__, peaceful ones. It is in the classroom that caring and respect for each person empowers children to take a step __6__ toward becoming peacemakers. Fortunately, educators have access to many online resources that are __7__ useful when helping children along the path to peace. The Young Peacemakers Club, started in 1992, provides a Website with resources for teachers and ___8__ on starting a Kindness Campaign. The World Centers of Compassion for Children International call attention to children’s rights and how to help the __9__ of war. Starting a Peacemakers’ Club is a praiseworthy venture for a class and one that could spread to other classrooms and ideally affect the culture of the __10__school. acting especially projects assuming forward respectively comprehensive images role cooperative information technology entire offers victims

Subjects 165 second-year students, 100 male and 65 female, majoring in Business Administration, Architecture, Software Design, Electronic Engineering and Chemistry at Shenzhen University, P. R. China Procedure The subjects were randomly assigned to the two versions of cloze —— the odd rows (Group A 1) took Normal Test One and the even rows (Group B 1) completed Scrambled Test One. Proficiency Exam * Administrated a week after the cloze tests * Check the equality of the two groups * Divide the subjects into 3 proficiency groups * Listening, Vocabulary and Structure, Reading, Translation and Essay writing. * Reliability value (Cronbach’s Alpha) -- 0. 78.

Table 1 Proficiency Exam Scores for Groups A 1 and B 1 N Mean SD Group A 1 87 63. 03 12. 03 Group B 1 78 61. 29 10. 54 t p . 983 . 327 Table 2 Proficiency Exam Scores for Three Proficiency Groups N Mean SD Group 1 58 74. 17 5. 93 Middle Group 1 49 61. 88 2. 86 Low 58 50. 53 6. 64 165 62. 21 11. 35 High Group 1 Total F 267. 44 p. 000 pos hoc tests: significant difference in each of the three pairs (. 000).

Results and discussion Question 1: Do subjects use text-level information in completing the banked cloze test? Table 3 Mean Scores in Normal and Scrambled Conditions N Mean SD Group A 1 87 6. 30 2. 61 Group B 1 78 5. 53 2. 53 t p 1. 962 . 052

Question 2: What types of information are used to answer each item? Table 4 Facility Index in Normal and Scrambled Conditions Facility Index IF Difference t p Normal (N=87) Scrambled (N=78) projects . 79 . 62 +. 17 2. 52 . 01 role . 74 . 53 +. 21 2. 85 . 01 acting . 66 . 56 +. 10 1. 19 . 23 offers . 92 . 95 -. 03 -. 75 . 46 cooperative . 36 . 00 -. 04 . 97 forward . 47 . 37 +. 10 1. 22 especially . 72 . 00 -. 03 . 98 information . 34 . 29 +. 05 . 63 . 53 victims . 77 . 68 +. 09 1. 35 . 18 entire . 53 . 45 +. 08 1. 10 . 28

Question 3: Does English proficiency affect their use of text-level information? PH=. 02 <. 05 PL=. 08>. 05

PH=. 04 <. 05 PL=. 08 >. 05 PM=. 08 >. 05

4. Study Two Purpose Expand the generalizability of the research, same issues explored Materials Normal Test Two *Reading in Depth Section, College English Test Band 4, June, 2008. *242 words *Chronological account / different type of text *10 contents words (4 verbs, 3 nouns, 2 adjectives and 1 adverbs) deleted Scrambled Test Two * Produced by rearranging the sentences in Normal Test Two * Inter-sentential connections were meant to be thoroughly disrupted

Subjects 141 second-year students, 108 male and 31 female, majoring in Information Technology, Civil Engineering and Maths at Shenzhen University, P. R. China Procedure The subjects were randomly assigned to the two versions of cloze —— the odd rows (Group A 2 ) took Normal Test Two and the even rows (Group B 2) completed Scrambled Test Two. Proficiency Exam * Administrated a week after the cloze tests * Applied as a criterion to evaluate their English competence * Listening, Vocabulary and Structure, Reading, Translation and Essay writing. * Reliability value (Cronbach’s Alpha) -- 0. 77

Table 5 Proficiency Exam Scores for Groups A 2 and B 2 N Mean SD Group A 2 69 67. 00 11. 07 Group B 2 72 66. 14 9. 93 t p . 487 . 627 Table 6 Proficiency Exam Scores for Three Proficiency Groups N Mean SD Group 2 54 77. 04 4. 89 Middle Group 2 36 66. 42 2. 66 Low 51 55. 57 5. 91 141 66. 56 10. 47 High Group 2 Total F p 255. 947. 000 pos hoc tests: significant difference between each of the three pairs (. 000).

Results and discussion Question 1 Do subjects use text-level information in completing the banked cloze test? Table 7 Mean Scores in Normal and Scrambled Conditions N Mean SD Group A 2 69 5. 55 2. 10 Group B 2 72 4. 39 2. 03 t p 3. 341 . 001 What has brought out the different alpha values in the two studies? (Study One: MGroup. A 1=6. 30; MGroup. B 1= 5. 53; t=1. 962, p>. 05) Text type? Cloze passage in Study One was largely an account of facts and did not exhibit much top-level constraint on the order in which propositions were presented while the passage in the Study Two was a chronological account of a writer’s experience, thus more sensitive to sentence reordering.

Question 2 Does English proficiency affect the use of text level clues? Diff=1. 49 PH=. 003 <. 05 Diff=0. 98 PM=. 140 >. 05 Diff=0. 77 PL=. 234 >. 05 As subjects become more proficient in English, they are better able to benefit from discourse constraints ranging across sentence boundaries.

Question 3 What types of information are used to answer each item? Table 8 Facility Index in Normal and Scrambled Conditions Facility Index IF Difference t p Normal (N=69) Scrambled (N=72) accomplish . 54 . 53 +. 01 . 024 . 981 limited . 84 . 00 . 080 . 937 scary . 25 . 10 +. 15 2. 399 . 018 totally . 74 . 75 -. 01 -. 194 . 846 interviews . 64 . 44 +. 20 2. 412 . 017 claim . 38 . 27 +. 11 1. 304 . 194 advanced . 58 . 56 +. 02 . 216 . 829 balloon . 81 . 56 +. 25 3. 320 . 001 moments . 52 . 12 +. 40 5. 541 . 000 regret . 26 . 23 +. 03 . 384 . 701

5. Conclusion There may be considerable variation in the effectiveness of banked cloze in assessing discourse level abilities. Different cloze items carry with them different amount or level of information. More proficient test-takers have better access to discourse level clues than the less proficient. And a well-ordered narrative/chronological text could function better than a loosely connected encyclopedic description which simply lists facts. Therefore it can be concluded that tester/researcher intervention is necessary for developing sound banked cloze tests.

Limitations § Only two samples of banked cloze tests § Convenience sampling of subjects § Subjects in the two separate studies – equivalent in their English competence? Group 1 Group 2 Test one Test Two Normal Scrambled Normal

Thank you very much!