PISA Programme for International Student Assessment PISA team

PISA Programme for International Student Assessment PISA team Department of Education – Ghent University – Belgium Beijing – July 24 -25, 2009 http: //allserv. ugent. be/~mvalcke/CV/CVMVA. htm Martin. Valcke@ugent. be (Partly based on Schleicher, A. , 2006)

Structure • • Starting grounds Objectives Samples – Population Quality benchmarks Framework Typical assessment approach National Centres

PISA: starting grounds • OECD: Organisation for Economical Organisation and Development • Basis of PISA: original work of OECD related to statistics and indicators about education – late 1980 s – voluntary contributions – Member engagement through networks

PISA: starting grounds • Network on educational outcomes – Proposal to study educational outcomes – Formally started with 11 members in 1996 – Expanded in 1997 • New name PISA • Members bear costs and risks – Number of countries grows every PISA edition: 2000, 2003, 2006, 2009, …

PISA 2000 OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Spanje, Zweden, Zwitserland, Verenigd Koninkrijk, Verenigde Staten Partnerlanden (niet-OESO): Albanië, Argentinië, Brazilië, Bulgarije, Chili, Hong Kong-China, Indonesië, Israël, Letland, Voormalige Joegoslavische Republiek Macedonië, Peru, Roemenië, Rusland, Thailand

PISA 2003 OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije, Verenigd Koninkrijk, Verenigde Staten Partnerlanden (niet-OESO): Brazilië, Hong Kong-China, Indonesië, Letland, Macao-China, Rusland, Servië & Montenegro, Thailand, Tunesië, Urugay

PISA 2006 OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Luxemburg, Mexico, Nederland, Nieuw. Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije, Verenigd Koninkrijk, Verenigde Staten Partnerlanden (niet-OESO): Argentinië, Azerbeidzjan, Brazilië, Bulgarije, Chili, Colombia, Kroatië, Estland, Hong Kong-China, Indonesië, Israël, Jordanië, Kyrgyzstan, Letland, Liechtenstein, Litouwen, Macao-China, Qatar, Montenegro, Servië, Roemenië, Rusland, Slovenië, Taipei, Thailand, Tunesië, Uruguay

PISA country participation OECD countries participating from PISA 2000 OECD countries participating from PISA from 2003 PISA 2000 (32 countries) OECD partner countries participating from PISA 2000 PISA 2003 (41 countries) OECD partner countries participating from PISA 2003 PISA 2006 (57 countries) OECD partner countries participating from PISA 2006

PISA: Objectives • A three-yearly global educational assessment • What did they learn? • Performance of 15 -year-olds • key subject areas, and a range of educational outcomes • Additionally: • student attitudes to learning, self efficacy beliefs, and learning strategies • contextual data from students, schools, parents and systems: policy levers See further

PISA: Objectives • Comparing performance within and between countries • Cross-cultural study • Central concept: LITERACY • Mathematical literacy • Scientific literacy • Reading literacy

PISA: Objectives Example: scientific literacy Is defined in terms of an individual’s: • Scientific knowledge and use of that knowledge to… … identify scientific issues, … explain scientific phenomena, and … draw evidence-based conclusions about science-related issues • Understanding of the characteristic features of science as a form of human knowledge and enquiry • Awareness of how science and technology shape our material, intellectual and cultural environments • Willingness to engage with science-related issues.

PISA: Population versus samples • Population of 15 year old pupils • National samples l l l Representative samples between 3, 500 - 50, 000 pupils Most federal countries: regional samples e. g. , Flanders versus Wallonia within Belgium PISA covers roughly 90% of the world economy.

Flanders Dutch speaking commmunity Belgium Federal state (10, 511. 382 inhabitants) Wallonia French speaking commmunity German speaking commmunity 13

PISA sampling requirements • Population: all 15 -year-olds in school – excludes 15 -year-olds out of school – includes 15 -year-olds in special education institutions – could exclude up to 5% of 15 -year-olds in school • difficult to reach (e. g. remote schools) • non-participation – few countries fail to reach required sample in 2003 • NZ (5. 1%), Denmark (5. 3%), UK (5. 4%), Canada (6. 8%), Spain (7. 3%), US (7. 3%) • Sample – – minimum of 150 schools per country two random samples: schools and replacement schools if school declines, replacement school is invited requirements set by countries

PISA: Networks • International expertise network building on the participating countries… – Instruments ~ input of > 40 countries (see next presentation) – Cross-national and cross-cultural validity – Analysis of results • International, natiopnal, regional analyses and reports • Country reviews • Consortium of research institutions ACER, CITO, ETS, NIER, WESTAT • Coordinated by OECD and international organisations

PISA: Objectives • Focus on performance in subject areas: – Languages: Reading literacy • Using, interpreting and reflecting on written material. – Mathematics: Mathematical literacy • Recognising problems that can be solved mathematically, representing them mathematically, solving them. – Sciences: Scientific literacy • Identifying scientific questions, recognising what counts as scientific evidence, using evidence to draw conclusions about the natural world.

PISA Objectives: cycles 2000 2003 2006 2009 2012 Reading Reading Maths Maths Science Science

PISA: Objectives • Focus on performance in additional domains subject areas: – – 2000: Problem Solving 2003: ICT literacy 2006: Attitudes towards science 2009: ERA

PISA key quality benchmarks • Overall performance of education systems • Equity in the distribution of learning opportunities – Measured by the impact students’ and schools’ socio-economic background has on performance… … not merely by the distribution of learning outcomes • Consistency of performance standards across schools • Gender differences • Foundations for lifelong learning

PISA 2000 2003 2006 2009 2012 Reading Reading Maths Maths Science Science ICT literacy Attitudes towards sciences ERA Problem solving

PISA framework Outputs and Outcomes impact of learning Quality and distribution of knowledge & skills

PISA framework Student background variables Mediating variables Literacy

PISA framework: complex interplay variables Student background variables Mediating variables System level Class-school level Student level Literacy

PISA framework Outputs and Outcomes impact of learning Quality and distribution of knowledge & skills Level B Instructional settings Quality of instructional delivery Level C Schools, other institutions Output and performance of institutions Social & economic outcomes of education

Student performance High Low Disadvantage PISA Index of social background Advantage

PISA framework Domain 2 Outputs and Outcomes Policy Levers impact of learning shape educational outcomes Quality and distribution of knowledge & skills Individual attitudes, engagement and behaviour Level B Instructional settings Quality of instructional delivery Teaching, learning practices and classroom climate Level C Schools, other institutions Output and performance of institutions The school learning environment Social & economic outcomes of education Structures, resource allocation and policies

PISA framework Domain 2 Domain 3 Policy Levers Antecedents impact of learning shape educational outcomes contextualise or constrain ed policy Quality and distribution of knowledge & skills Individual attitudes, engagement and behaviour Socio-economic background of learners Instructional settings Quality of instructional delivery Teaching, learning practices and classroom climate Student learning, teacher working conditions Schools, other institutions Output and performance of institutions The school learning environment Community and school characteristics Social & economic outcomes of education Structures, resource allocation and policies National education, social & economic context Outputs and Outcomes Level B Level C

Typical PISA assessment • Information collection – From students • • 3½ hours of main domain assessment 1 hour in relation to other subdomain 2 hours on paper-and-pencil tasks (subset of all questions) ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation – From school principals • Questionnaire (school demography, learning environment quality) • Indirect assssment of classroom variables (teacher, class) BEWARE!! Only adequate if grade based sampling has been applied

PISA National Centre • Linking with international consortium • Implementation of framework ~national level • Reporting to consortium • Representation during international meetings

PISA International • • Strong prescriptive framework Framework, timing, procedures, tools National data gathering International processing of data Priority in international release of results • National (regional) processing of data next priority level • Secondary analysis of data: data available

New developments • Towards electronic assessment • 2009: first full scale trial – ERA Electronic Reading Assessment

PISA Programme for International Student Assessment PISA team Department of Education – Ghent University – Belgium Beijing – July 24 -25, 2009 http: //allserv. ugent. be/~mvalcke/CV/CVMVA. htm Martin. Valcke@ugent. be (Partly based on Schleicher, A. , 2006)