PISA for teachers Interpreting and using information from

PISA for teachers Interpreting and using information from an international reading assessment in the classroom ACER Research Conference Perth, 16 -18 August, 2009 Juliette Mendelovits Dara Searle Tom Lumley

Outline • Why do countries participate in international studies of educational achievement? • Australia’s performance in PISA reading, 2000 to 2006 – What do our results tell us about the way Australian 15 -year-old read? – What can teachers do to improve reading performance? • New directions in PISA reading 2009: Electronic reading: – How has PISA conceptualised electronic reading? – How can this conceptualisation be used at the classroom level?

Why do countries participate in international studies of educational achievement? • To collect national-level data when there is no other mechanism for doing so • To compare themselves with other countries • To be involved in – and help to shape - new developments in education

AUSTRALIA’S PERFORMANCE IN PISA READING, 2000 TO 2006

In PISA 2000 Australia Ranked 4 th out of the 31 participating countries 5

Australia’s result for PISA Reading 2000 • Mean= 528 – OECD average =500 • Standard deviation =102 – OECD average = 100

Australia’s results for PISA Reading 2000 Subgroup performance • Gender – Difference between girls and boys (34 points) was about the same as the OECD average (32 points). • Indigenous students – Performed about 80 PISA scale score points below nonindigenous students – the equivalent of about two years of schooling • High performing students – The top performing 5% of students in Australia performed as well as any other countries’ top 5%

PISA survey cycle

PISA Reading 2000 – 2006 • Between PISA 2000 and PISA 2006: Ø 7 countries showed an improvement in their results Ø 15 countries showed no change in their reading literacy performance Ø 14 countries showed a decline in their results … Øincluding Australia

Australia’s result for PISA Reading 2006 • Mean= 513 – OECD average =492 • Standard deviation =94 – OECD average = 99

PISA Reading in Australia 2000 – 2006 • Australia’s mean score in reading declined by 15 scale score points – the equivalent of about 4 months of schooling • Results declined for all states and territories • Maybe it was the sample? – But Australia showed no significant change in performance in Mathematical literacy between PISA 2003 (when Mathematics was the major domain) and PISA 2006

PISA Reading in Australia 2000 – 2006 Subgroup performance in reading • Gender – Both girls and boys declined in performance (the gap remained about the same) • Indigenous students – Indigenous performance declined from 2000 to 2006, but not significantly – http: //www. acer. edu. au/ozpisa/reports. html • High performing students – There was a significant decline in performance of the top 25% of Australian students • already observable in 2003, worsened in 2006 • top 5% declined the most

What happened and what can we do about it? • The decline is NOT attributable to: – the sample (compare maths) – differences between states and territories – a gender difference problem – a decline in indigenous performance • Maybe it can be attributed to: – disproportionate focus on improvement of the least proficient (high performers showed the biggest decline) – the way reading is taught in Australia …

PISA’s definition of reading Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.

PISA uses five task characteristics in the construction of all reading literacy tasks • • • Context Medium Environments Text format – text type Aspect

Reported Subscales for PISA Reading 2000 Text format – Continuous – Non-continuous Aspects – Access and retrieve – Integrate and interpret – Reflect and evaluate

Text format • Continuous – Information presented in sentences and paragraphs • • • Argument Description Exposition Instruction Narration • Non-continuous – Information presented in formats other than prose, for example • • • tables, graphs, diagrams maps forms

In PISA 2000, Australia performed better on non-continuous texts than continuous texts Our weakest performance, in terms of text type, was on narrative and argumentative texts.

Aspects • Access and retrieve • Integrate and interpret • Reflect and evaluate

In PISA 2000, Australia performed best on access and retrieve tasks, and worst on reflect and evaluate tasks

Comparing Australia’s results with those of other English speaking countries in PISA 2000 21

Aspect means for English-speaking countries

Sample reflect and evaluate item, narrative text The Motorcycle Have you ever woken up feeling that something was wrong? It was a day like that for me. I sat up in bed. A little later I opened the curtains. It was terrible weather – the rain was pouring down. Then I looked down into the yard. Why does the writer begin the story with a question? Yes! There it was – the motorcycle. A. Because the writer wants to know the It was just as wrecked as last night. answer. And my leg was starting to hurt. B. To involve the reader in the story. C. Because the question is hard to answer. D. To remind the reader that this kind of experience is rare.

Sample reflect and evaluate item, narrative text Macondo Dazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom-toom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in theatre with the lion-head ticket windows, for a character who had died and was buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the Do you agree with the final judgement of the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine of illusions people of Macondo about the value of the that did not merit the emotional outburst of the audience. With that movies? discouraging explanation many felt that they had been the victims of Explain your answer by comparing your some new and showy gypsy business and they decided not to return to the movies, considering that they already had too many troubles of their attitude to the movies with theirs. own to weep over the acted-out misfortunes of imaginary beings.

Example answers for Macondo

Sample task from an argument text Democracy in Athens One purpose of the speech in part B was to PART A honour soldiers who fell in the first year of Thucydides was a historian and military man who lived in the fifth century the Peloponnesian War. BC, during the Classical Greek period. He was born in Athens. During the Peloponnesian War (431 BC to 404 BC) between Athens and Sparta he was in command of a fleet whose mission was to protect the What was ANOTHER purpose of this city of Amphipolis in Thrace. … speech? PART B Thucydides attributes to Pericles (fifth century BC), the Athenian ruler, the following speech in honour of the soldiers who fell in the first year of the Peloponnesian War. Our system of government does not copy the laws of neighbouring states; we are rather a pattern to others than imitators ourselves. Our system is called democracy, since its administration depends on the many instead of the few. Our laws afford equal rights to all in their private affairs, whereas the prestige in public life depends on merit rather than on social class. …

Add democracy in Athens?

Based on information from PISA, what can teachers do to improve reading performance of Australian students? • Reconsider approaches to reflective and evaluative reading; • Change the emphasis of what is done with continuous texts, in particular narratives and argument texts; and • Make particular efforts to challenge the most able students.

Why do countries participate in international studies of educational achievement? • To collect national-level data when there is no other mechanism for doing so • To compare themselves with other countries • To be involved in – and help to shape - new developments in education

NEW DIRECTIONS IN PISA READING: ELECTRONIC READING:

Why introduce an Electronic Reading Assessment (ERA)? • As of 2007, almost one and a half billion people were reading on line • Between 2000 and 2007, Internet use doubled in OECD countries, and more than trebled worldwide

Electronic reading is not ICT literacy Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society. (PISA definition of reading literacy) Electronic reading is not just reading print text on a computer screen

What distinguishes electronic reading from print reading? § Different kinds of texts (more multiple texts, more bits and pieces, more opportunity to interact with the text) § More onus on the reader to evaluate the text § More onus on the reader to select and construct the text because § § There’s usually no pre-determined order in which a text should be read It is a more abstract and unknown navigation space

Analogies … In the print medium Page of a go In the electronic medium Page of a book Web page A book A web site Bibliography, references, “further reading” Links on a web site to other sites A very big library The Internet

In PISA, electronic reading is conceptualised in terms of two components • Text processing • Navigation

Relationship between navigation and text processing for ERA tasks 36

ERA Sample Items http: //erasq. acer. edu. au

Relationship between navigation and text processing for ERA tasks E 022 Q 09: Write a reply to Mischa. In your reply, answer her question… E 022 Q 08: Which suggestion does Doctor Nauckunaite make? E 022 Q 01: Who wrote the first reply… E 022 Q 04: Which writer would disagree… 38

Relationship between navigation and text processing for ERA tasks E 022 Q 09: Write a reply to Mischa. In your reply, answer her question… E 022 Q 08: Which suggestion does Doctor Nauckunaite make? E 022 Q 01: Who wrote the first reply… E 022 Q 04: Which writer would disagree… 39

Based on information from PISA, what can teachers do to improve electronic reading performance? 1) Different kinds of texts Teachers could check that students are exposed to a range of text forms from the digital medium 2) More onus on the reader to evaluate the text Teachers could refer to the framework descriptions and the items that reflect them as models of critical reading in the electronic medium. 3) More onus on the reader to select and construct the text Teachers could inspect this range of tasks to help construct a sequence of lessons on classifying and mastering different navigation techniques ALONG WITH the thinking (reflection and evaluation) that’s necessary to find information efficiently

An assessment system like PISA can help teachers to think about • The kinds of texts that students are taught • The kinds of approaches to reading (PISA’s ‘aspects’) that can be covered • Ways of attending to the full distribution of student achievement

The PISA Proficiency Scale 10% 22% Australian Average OECD Average 29% Reading Literacy Level 5 Evaluating information and building hypotheses Drawing on specialised knowledge Accommodating concepts contrary to expectations 22% 12% 6 % Reading Literacy Level 1 Recognise main theme on a familiar topic, make simple connections Below Level 1 These students have not acquired the skills to use reading for effective participation in society 42

The PISA Proficiency Scale 10% 22% Australian Average OECD Average 29% Reading Literacy Level 5 Evaluating information and building hypotheses Drawing on specialised knowledge Accommodating concepts contrary to expectations 22% 12% 6 % Reading Literacy Level 1 Recognise main theme on a familiar topic, make simple connections Below Level 1 These students have not acquired the skills to use reading for effective participation in society 43

Thank you For more information about PISA go to www. pisa. oecd. org

Text formats … • Continuous • Non-continuous • Mixed • Multiple further categorised by text types • Narration • Exposition • Description • Argumentation • Instruction 45