PISA International Conference Reading Performance of Hong Kongs

PISA International Conference

Reading Performance of Hong Kong’s 15 -Year-Old Students in PISA

Contents I. Design of Reading Assessment n The construct of reading literacy n Design of PISA assessment tasks II. The findings n The overall and specific performance of HK students on reading literacy n III. Some Observations

The Construct of Reading Literacy in PISA

The Construct of Reading Literacy in PISA “the ability to understand, use and reflect on written texts in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate effectively in society” (OECD, 1999)

Reading Literacy in PISA 3 dimensions of reading Process of Reading Content of Reading Context of Reading

The Process - Five aspects of reading Retrieving information n Broad understanding Interpreting n Developing an interpretation n Reflecting on content Reflecting and Evaluating content and form n Reflecting on form n

The Content – 2 categories, 11 types Text Continuous Descriptive Narrative Expositive Argumentative Injunctive Non-Continuous Charts and Graphs Tables Diagrams Maps Forms Advertisements

The Context Personal n Public n Occupational n Educational n

The Construct of Reading Literacy for PISA Process (retrieving, interpreting, reflecting) Context (Personal, Public, Occupation, Education) Content (Format, Type)

The Design of PISA Literacy Assessment

The Hong Kong Sample PISA sampling procedure closely followed n 140 schools n 4405 15 -year-old students n

The assessment materials 37 stimulus texts n 141 reading tasks n 9 equal sets n Chinese version verified by PISA n

Formats of Response Multiple choice questions n Complex multiple-choice items n Closed-constructed responses n Short responses n Open-ended responses n

Reporting of Reading Literacy Performance n n Four different proficiency scales Retrieving information sub-scale Interpreting text sub-scale Reflecting on and evaluating text sub-scale Combined scale These scores then mapped on to a five-level proficiency scale

Descriptors of skills demonstrated at each level of the three reading proficiency sub-scales (Appendix 3) Proficiency Level Retrieving Information Interpreting Reflecting Level 5 Locate and possibly sequence or combine multiple pieces of deeply embedded information, some of which may be outside the main body of the text. Infer which information in the text is relevant to the task. Deal with highly plausible and/or extensive competing information. Either construe the meaning of nuanced language or demonstrate a full and detailed understanding of a text. Critically evaluate or hypothesise, drawing on specialised knowledge. Deal with concepts that are contrary to expectations and draw on a deep understanding of long or complex texts. Level 3 Locate, and in some cases recognize the relationship between pieces of information, each of which may need to meet multiple criteria. Deal with prominent competing information. Integrate several parts of a text in order to identify a main idea, understand a relationship or construe the meaning of a word or phrase. Compare, contrast or categorize taking many criteria into account. Deal with competing information. Make connections or comparisons, give explanations, or evaluate a feature of text. Demonstrate a detailed understanding of the text in relation to familiar, everyday knowledge, or draw on less common knowledge. Level 1 Locate one or more independent pieces of explicitly stated information, typically meeting a single criterion, with little or no competing information in the text. Recognize the main theme or author's purpose in a text about a familiar topic, when the required information in the text is not prominent. Make a simple connection between information in the text and common, everyday knowledge.

Table 5: Reading literacy score range of the reading proficiency levels Reading Literacy Scores Proficiency Level 5 Proficiency Level 4 Proficiency Level 3 More than 625 553 – 625 481 – 552 Proficiency Level 2 408 – 480 Proficiency Level 1 335 – 407 Proficiency Level Below 1 Below 335

The testing material – A sample item Graffiti Question 1: The purpose of each of these letters is to: A explain what graffiti is. B present an opinion about graffiti. C demonstrate the popularity of graffiti. D tell people how much is spent removing graffiti. Answer: B Situation: Public Text format: Continuous Aspect: Interpreting texts Level: 2 PISA scale score: 421

HK Students’ Performance Overall

Table 6: Combined Reading Literacy Mean scores in reading literacy of participating countries/regions ＊significant difference ○ insignificant difference Mean S. E. Finland 546 (2. 6) ＊ Canada 534 (1. 6) ○ New Zealand 529 (2. 8) ○ Australia 528 (3. 5) ○ Ireland 527 (3. 2) ○ Hong Kong, China 525 (2. 9) - Korea 525 (2. 4) ○ United Kingdom 523 (2. 6) ○ Japan 522 (5. 2) ○ Sweden 516 (2. 2) ○ Austria 507 (2. 4) ＊

Fig. 2 Percentage of students performing at each of the proficiency levels on the combined reading literacy scale At Level 5 At Level 4 At Level 3 At Level 2 At Level 1 Below Level 1

Combined reading score Fig. 3 A comparison of students performance on the combined reading literacy scale 700 650 600 550 500 450 400 350 300 Hong Kong OECD average 0 10 20 30 40 50 60 70 80 90 100 Percentile

Descriptors of performance by students at Level 1 and Below Level 1 n Level 1: “capable of completing only the least complex reading tasks, such as locating a single piece of information, identifying the main theme of a text or making a simple connection with everyday knowledge. ” n Below Level 1 : “not capable of the most basic type of reading that PISA seeks to measure……Such students have serious difficulties in using reading literacy as an effective tool to advance and extend their skills in other areas. ”

Summary of HK students’ overall performance on reading literacy HK has a relatively high performance (525, Proficiency Level 3, 6 th) in reading literacy comparing with other countries n Comparing with other countries, HK has a relatively small percentage (10%) of very proficient readers and poor readers n The high rank of HK is largely due to the large percentage of students achieving Level 3 (33%) & 4 (31%) n

HK Students’ Performance Specific

Table 7: Proportion of students at each proficiency level of the reading sub-scales Proficiency level Retrieving Information Interpreting Reflecting and Evaluation HK OECD Difference Below 1 4% 8% -4% 2% 6% -4% 3% 7% -4% Level 1 8% 12% -4% 7% 12% -5% 6% 11% -5% Level 2 19% 21% -2% 18% 22% -4% 15% 21% -6% Sub-total (Level 2 and below) 31% 41% Level 3 30% 26% Level 4 28% Level 5 12% Sub-total (Level 3 and above) -10% -13% 27% 40% +4% 34% 28% 21% +7% 30% 12% 0% 8% 24% 39% +6% 29% 28% +1% 22% +8% 32% 23% +8% 10% -2% 15% 11% +4% +11% 70% 59% Note: There may be rounding errors. -15% +12% 72% 60% +14% 76% 62%

Summary of HK students’ performance on the reading sub-scales n n HK students generally do better than OECD countries on the sub-scales HK students maintain their good performance on “reflecting and evaluation” HK high-achieving students gradually improve their performance from retrieving to evaluating, while low achievers showed a reversed pattern of results HK students’ performances show a greater disparity across the sub-scales than that of the OECD countries

Table 8: Mean percentage scores across different text formats Mean Percentage Scores Continuous texts Non- Continuous texts No of items HK OECD* 85 42 66 64 61 61 *2 items were deleted from the Hong Kong data set for subsequent analysis. This OECD average is calculated based on 127 corresponding items.

Table 9: Mean percentage scores across different text types Mean Percentage Score (%) TEXT TYPE No. of tasks Hong Kong OECD Narrative 17 64 61 Argumentative/Persuasive 17 69 60 Expository 31 68 63 Descriptive 11 63 60 Injunctive 9 62 58 Advertisements 1 20 29 Forms 5 69 62 Maps 4 75 75 Charts/Graphs 15 69 68 Diagrams 5 62 56 Tables 12 57 51 Continuous text Non-continuous text

Table 10: Mean percentage scores across different contexts of use Mean Percentage Score (%) Context No. of items HK OECD Educational 36 68 64 Occupational 19 69 58 Personal 25 65 60 Public 47 63 61

Conclusion and Discussion

n n n n Comparatively speaking, HK lags behind in the number of very proficient readers. HK students are relatively good at high order reading skills. HK students are particularly good at argumentative text, but weak at advertisement. HK students perform better on “educational”, “occupational” than on “personal” and “public” context of reading. HK education is successful in producing a majority of students with an above-average reading literacy. Are the results contradictory to our perception? How good do we want our students to be?