New Survey Questionnaire Indicators in PISA and NAEP

New Survey Questionnaire Indicators in PISA and NAEP Jonas Bertling Director, NAEP Survey Questionnaires, ETS RWJF & ETS Workshop Understanding the opportunities for collaboration between the health and education sectors in a culture of health Princeton, May 5, 2016

Main talking points today • Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes. • PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in 2015. PISA 2018 will include an optional well-being questionnaire. • NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 36319 2

Main talking points today (continued) • NAEP 2017 contextual modules can also be mapped to PISA 2018 draft well-being framework. • PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictor side only. • Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 3

Changing role of large-scale assessments How are students in the Nation doing in terms of other outcomes than achievement? How do students approach learning situations and utilize learning opportunities? How do learning opportunities differ across the Nation? How do subgroups perform? J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 4

Shift in policy interest Measure contextual variables Measure constructs of own interest • Describe test-taker populations • Report additional outcomes • Improve estimates of achievement (conditioning model) • Measure trend • Compare subgroups • Evaluate policies • Accountability Figure from Bertling, Borgonovi, & Almonte (2016) J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 5

J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 6

Survey questionnaires data put achievement results into context … … and can add value to large-scale assessments by capturing important additional variables. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

PISA Programme for International Student Assessment Organisation for Economic Co-operation and Development (OECD) 8

PISA broadened the range of constructs captured with survey questionnaires since the first cycle in 2000 • Since its early cycles, PISA has included questions capturing students’ attitudes and sense of belonging to school • PISA 2012 introduced new domain-general student attitudes and behaviors (e. g. , perseverance, openness for problem solving) • PISA 2015 added additional noncognitive outcomes also covering health and well-being • This trend continues with PISA 2018 Figure from Bertling, Borgonovi, & Almonte (2016) J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

PISA 2015 Questionnaire Framework Figure from OECD (2016) J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

Well-being-related constructs in PISA 2015 • General life satisfaction • Participation in physical education classes at school • Engagement in moderate and vigorous physical exercise • School-related well-being (sense of belonging, bullying, test anxiety) A larger number of additional health- and well-being-related questions (e. g. , BMI, dietary habits, enjoyment of physical activity, etc. ) were piloted but not taken to operational largely due to timing constraints. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 11

Well-being questionnaire in PISA 2018 • OECD is planning an optional questionnaire devoted to student well-being for PISA 2018 in addition to other noncognitive factors captured with the regular student questionnaires. DRAFT* • Developed draft framework for PISA 2018 Module *Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject to all the normal OECD copyright provisions, and should clearly be acknowledged with the OECD as owner and author. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 12

NAEP National Assessment of Educational Progress National Center for Education Statistics (NCES) 13

Innovations underway for the NAEP Survey Questionnaires Expand constructs to include more I. contextual factors critical for student achievement Improve reporting of contextual factors II. for more robustness & policy relevance Improve pre-testing and questionnaire III. design for digital delivery to maximize validity and keep student burden low J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

NAEP Contextual Variables (SURVEY QUESTIONNAIRES) BUT: The term “noncognitive” is not intended to be interpreted literally. All aspects of human thought and behavior are related to cognition! “Noncognitive” Student Factors: Skills, strategies, attitudes and behaviors that are distinct from content knowledge and academic skills (e. g. , motivational and selfregulatory variables) Opportunity-to-Learn (OTL) Factors: Students’ exposure to opportunities to acquire relevant knowledge and skills, both at school and outside of school. Domaingeneral factors (“Core”) Subjectspecific factors “Noncognitive” Student Factors Cognitive Ability* Achievement OTL Outside of School OTL at School Basic Demographics *not measured with the NAEP survey questionnaires (COGNITIVE ASSESSMENT) Balanced reporting approach based on indices and stand-alone questions J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

Implementation for NAEP 2017 Core, Reading, Writing and Mathematics: Item Development Pre-testing in Cognitive Labs Large-scale Pilot Testing Operational Administration 2014 -15 2016 2017 Domaingeneral (Core) Noncogs Subjectspecific “Noncognitive” 2 1 2 Student Factors 1 3 OTL Outside of School 4 3 Outside of School OTL at School 6 OTL At School 4 5 5 7 Basic Demographics 5 Core Modules Additional Subjectspecific Modules 1. Grit/Perseverance (including self-control) 2. Desire for Learning 1. Self-Efficacy/Confidence 2. Achievement Goals 3. Interest & Enjoyment 3. Components of SES 4. Technology Use 5. School Climate 4. Out-of-School Activities 5. Resources for Learning and Instruction 6. Organization of Instruction 16 7. Teacher Preparation J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

Mapping NAEP to PISA 2018 WB Framework DRAFT* Grit/Perseverance, Desire for Learning, Self-Efficacy School Climate SES *Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject to all the normal OECD copyright provisions, should clearly be acknowledged with the OECD as owner and author. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 17

Difference between PISA and NAEP approaches Additional Outcomes Achievement Predictors J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 18

Challenges 1. Measurement 2. Perceived Intrusiveness 3. Respondent Burden Figure from Bertling, Borgonovi, & Almonte (2016) J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 19

Summary • Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes. • PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in 2015. PISA 2018 will include an optional well-being questionnaire. • NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 20

Summary (continued) • NAEP 2017 contextual modules can also be mapped to PISA 2018 draft well-being framework. • PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictor side only. • Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden. J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 21

Thank you! Questions? Comments? jbertling@ets. org Twitter: @Jonas. Bertling J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 36319 22