PISA for Development Technical Strand 1 Contextual questionnaires

PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD

PISA for Development Contextual questionnaires q Purpose of PISA contextual questionnaires q How are they used? q How they are developed q What will PISA for Development seek to do? q Questions for discussion 2

PISA for Development Purpose of PISA contextual questionnaires ü Student and school questionnaires are part of the core assessment design of PISA ü Countries may also opt for additional questionnaires: - Parents, Teachers, Health, Time Use, ICT familiarity, Career… ü Student questionnaires: ~ 30 minute* stand-alone document (students respond after test questions) ü School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing) * Rotated questionnaires in 2012 to cover additional material

PISA for Development Purpose of PISA contextual questionnaires ü Contextualise the performance results in reading, mathematics and science* ü Allow analyses of equity issues across sub-groups and populations ü Provide indications of possible policy interventions ü Some information collected is used to scale cognitive assessment data ü Country-specific questions can be included for additional analysis * Additional domains such as financial literacy and problem-solving

PISA for Development Purpose of PISA contextual questionnaires ü Have been used in 74 countries/economies ü In more than 40 languages ü Publicly available PISA 2009 Student Questionnaire PISA 2012 School Questionnaire PISA 2009 School Questionnaire

PISA Contextual Questionnaires How they are used Levels of an education system This is the PISA Analytic Framework that guides data collection and analysis Outputs and Outcomes impact of learning Policy Levers Antecedents shape educational outcomes contextualise or constrained policy Individual learner Quality and distribution of knowledge & skills Individual attitudes, engagement and behaviour Socio-economic background of learners Instructional settings Quality of instructional delivery Teaching, learning practices and classroom climate Student learning, teacher working conditions (other institutions) Output and performance of institutions The learning environment at school Community and school characteristics Country or system Social & economic outcomes of education Structures, resource allocation and policies National educ. , social and economic context Schools

PISA Contextual Questionnaires How they are used This is how the PISA Analytic Framework is operationalised for data collection and analysis Main data collection Student Questionnaire, School Questionnaires and System Level* Outcomes Students Classrooms Schools Country Processes Antecedents Variables included in instruments: • General recurring variables (all cycles) • Domain recurring variables (main domain every 3 years) • Extension variables (specific cycles) • System-level data * Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career

PISA Contextual Questionnaires System-level data collection Standards and procedures for data collection in PISA These are examples of the types of information collected at the system level: • • • ü Questionnaires for non-OECD economies ü Panel of experts to complete some data collection Ratio: students / teaching staff Spending /student Sample of relevant areas to Instruction time (by age) address in developing contexts Teachers’ salaries (by education, experience) Evaluation & accountability Educational stratification • Within-country disparities Decision-making • Quality, supply and distribution of educational at different levels materials and resources School choice • (Extra-)tuition Changes in education • Parental support policy • Contractual arrangements Others (…) of school teaching staff 8

PISA Contextual Questionnaires School questionnaires This is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources • • • • School resources, policies and practices These are the constructs that will be reviewed as part of the technical work of PISA for Development Grades served Public-Private Funding sources Community type Enrollment (boys, girls) Average class size Non-native speakers Grouping/tracking Teachers, by qualifications Teacher shortages Physical building limitations Instructional materials Teacher attitudes, expectations, relations Computer/network/web access • • • Absenteeism, disruptions Teacher morale Student morale Teaching environment Time for special programmes Non-native speaker offerings Admissions requirements Assessment types Uses of assessment Assessment of teachers Decision making authority External decision makers 9

PISA Contextual Questionnaires Student questionnaires Student attitudes, behaviours and approaches to learning General and domain-related processes These are examples of the types of non-cognitive outcomes on which PISA collects information Sense of belonging at school Student-teacher relations at school Disciplinary climate in classes Teacher support in subject-specific classes Out-of-school activities (homework, etc. . . ) Self and domain-related cognitions Anxiety towards “study subject” Instrumental motivation to learn “study subject” Interest in and enjoyment of “study subject” Self-efficacy in “study subject” Self-concept in “study subject” Learning strategies Others Preference for competitive learning situations Preference for co-operative learning situations Control strategies Perseverance Health status, Time use (…) These are the constructs that will be reviewed as part of the technical work of PISA for Development 10

PISA Contextual Questionnaires Examples of analysis … How are questionnaire data used for relevant analysis and comparative policy insights? (a few examples from PISA results) 11

2003 12 Kyrgyzstan 600 2006 Indonesia Albania Georgia Tamil Nadu-India Himachal Pradesh-India 2009 Chinese Taipei Latvia Lithuania Russian Federation Costa Rica Serbia Bulgaria Uruguay Romania Venezuela-Miranda Thailand Malaysia Colombia Brazil Montenegro Mauritius Jordan Tunisia Argentina Kazakhstan Panama Peru Azerbaijan Reading performance Shanghai-China Hong Kong-China Singapore Liechtenstein Macao-China Croatia Dubai (UAE) Malta United Arab Emirates Trinidad and Tobago Qatar Examples of analysis Range of performance among non-OECD countries 2000 PISA Results for different years 550 500 450 400 350 300

Mexico Examples of analysis PISA 2009 Results School performance and socio-economic background School performance and students’ socio-economic background within schools Score 600 Student performance and schools’ socio-economic background Schools with a similar socio-economic profile Private school Public school in rural area Public school in urban area 550 OECD average 450 400 350 OECD average Student performance 500 300 250 200 -3 Disadvantage -2. 5 -2 -1. 5 -1 -0. 5 0 PISA Index of socio-economic background 0. 5 1 1. 5 Advantage 2

Kyrgyzstan Examples of analysis PISA 2009 Results School performance and socio-economic background School performance and students’ socio-economic background within schools Student performance and schools’ socio-economic background Score OECD average 600 550 OECD average Student performance 500 Private school Public school in rural area Public school in urban area 450 400 350 300 250 200 -3 Disadvantage -2. 5 -2 -1. 5 -1 -0. 5 0 0. 5 PISA Index of socio-economic background 1 1. 5 2 Advantage

Examples of analysis Policies and practices Policy R R System E School Equity Policy findings from contextual information Learning climate Policy briefs on these issues can be found in the PISA in Focus series Discipline Teacher behaviour Parental pressure Teacher-student relationships Dealing with heterogeneity Grade repetition Prevalence of tracking Expulsions Ability grouping (all subjects) Standards /accountability Nat. examination

to universal achievement Based on the analysis of Examples of analysis cognitive outcomes and Policy insights using contextual information Coherence A learning system information Goals, gateways, instructional systems Lessons from PISA from successful education systems (equitable and improvers) Resources where they yield most Incentives and accountability Capacity at point of delivery

PISA Contextual Questionnaires How they are developed l ‘Crowd sourcing’ and collaboration – PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies… … guided by governments on the basis of shared policy interests l Cross-national relevance and transferability of policy experiences – Emphasis on validity across cultures, languages and systems – Frameworks built on well-structured conceptual understanding of assessment areas and contextual factors l Continuous review and updating of assessment frameworks (cycles) – Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA – Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors 17

PISA Contextual Questionnaires How they are developed PISA 2012 Questionnaires • Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions) • School questionnaire: 33 questions (5 additional questions on financial education at school) • Parental questionnaire: 25 questions • ICT questionnaire: 12 questions • Education careers: 14 questions Linkages between cognitive and non -cognitive outcomes

PISA Contextual Questionnaires How they are developed Purpose of the PISA Field Trial (for Main Study) • Selection of cognitive test items to be included in the PISA Main Study instruments (e. g. country. Xcountry DIF, gender DIF) • Validation of psychometric equivalence of translated instruments (and adaptations from source versions) issues addressed • Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study • Determine cross-national validity of the questions in the questionnaire • Technical standards ensure that data from participating countries are internationally comparable

PISA Contextual Questionnaires What will PISA for Development seek to do? q Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results ü Adapt existing constructs (and indices and variables) ü Identify and introduce new constructs (indices and variables) ü Confirm and validate enhancements through field trials and main data collection in participating countries

PISA Contextual Questionnaires What will PISA for Development seek to do? Examples of variables and indices Student Questionnaire Section 1: ABOUT YOU Section 2: YOUR FAMILY AND YOUR HOME Section 3: YOUR READING ACTIVITIES Section 4: LEARNING TIME Section 5: YOUR SCHOOL Section 6: YOUR SCHOOL LESSONS AND SUBJECTS Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS Section 8: YOUR VIEWS ON <BROAD SCIENCE> Section 9: CAREERS AND <BROAD SCIENCE> Section 10: YOUR MATHEMATICS EXPERIENCES

PISA Contextual Questionnaires Student questionnaires Examples of variables and indices Student Questionnaire SECTION 2: YOUR FAMILY AND YOUR HOME 2009 -ST 08: Family structure 2009 -ST 09: Mother’s main job (ESCS, HISEI, BMMJ) 2009 -ST 10: Mother’s education (ESCS, HISCED, PARED, MISCED) 2009 -ST 11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED) 2009 -ST 12: Mother’s employment status (ESCS, HISEI, BMMJ) 2009 -ST 13: Father’s main job (ESCS, HISEI, BMFJ) 2009 -ST 14: Father’s education (ESCS, HISCED, PARED, FISCED) 2009 -ST 15: Father’s qualifications (ESCS, HISCED, PARED, FISCED) 2009 -ST 16: Father’s employment status (ESCS, HISEI, BFMJ) 2009 -ST 17: Country of birth for student and parents (IMMIG) 2009 -ST 18: Age at arrival in country of test 2009 -ST 19: Language spoken at home 2009 -ST 20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS) 2009 -ST 21: Family wealth (ESCS, HOMEPOS, WEALTH) 2009 -ST 22: Books in home (ESCS, HOMEPOS)

PISA Contextual Questionnaires Student questionnaires – examples of indices PISA index of Economic, Social and Cultural Status (ESCS) ü Comparability across countries (not a countryspecific income indicator) WEALTH Highest occupational status of parents HISEI Home possessions HOMEPOS Highest educational level of parents CULTPOSS PARED HEDRES # of books in home

PISA Contextual Questionnaires Student questionnaires – examples of indices PISA index of Economic, Social and Cultural Status (ESCS) Index of family wealth a room of their own, a link to the Internet, a dishwasher (treated as a country-specific Index of cultural possessions item), a DVD player, and three other classic literature, books of poetry and country-specific items (some items in WEALTH works of art (ST 20), and their responses on the number Index of home educational resources of cellular phones, televisions, computers, For example, availability of a desk, a quiet cars and the rooms with CULTPOSS place to study, a computer that students can a bath or shower (ST 21). use for schoolwork, educational software, HEDRES books to help students, technical reference books and dictionary (some items in ST 20) # of books in home

PISA Contextual Questionnaires Student questionnaires – examples of indices Self-efficacy in science and mathematics ü 2006 -ST 17: Self-efficacy in science (SCIEEFF) ü 2003 -ST 31: Self-efficacy in Stays the same in PISA 2012 mathematics (MATHEFF) Relationship between constructs, indices and variables (questions)

PISA Contextual Questionnaires Student questionnaires – examples of indices Self-efficacy in science ü 2006 -ST 17: Self-efficacy in science (SCIEEFF) 06 -ST 17 Q 42 How easy do you think it would be for you to perform the following tasks on your own? (Please darken only one circle in each row. ) I could do this easily a) Recognise the science question that underlies a newspaper report on a health issue ● 1 I would I could struggle do this to do this with a bit on my I couldn’t of effort own do this ● 2 ● 3 ● 4

PISA Contextual Questionnaires Student questionnaires – examples of indices Instrumental motivation to learn science ü 2006 -ST 35: Instrumental motivation to learn science (INSTSCIE) 06 -ST 35 Q 46 How much do you agree with the statements below? (Please darken only one circle in each row. ) a) b) Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on What I learn in my science Strongly agree Agree ● ● 1 2 Strongly Disagree disagree ● 3 ● 4

PISA for Development How will we do this? ü Identify constructs in need of adaptation and important constructs that are missing ü Obtain input and advice from experts in the field (grounded in developing-country contexts) ü Obtain input and guidance from participating countries ü Draw on existing work and empirical evidence ü Test the constructs (indices and variables) to ensure comparability with PISA scales

PISA for Development Technical Strand 1: Contextual questionnaires

PISA for Development Questions for discussion ü Are the intentions and rationale of this strand of work clear and understood? ü Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation? ü Where do you see the largest hurdle to overcome? ü What existing studies and evidence should contribute to this strand of work?