PISA for Development 1 st International Advisory Group

PISA for Development 1 st International Advisory Group Meeting Component Terms of Reference for the IC 27 – 28 May 2014 Paris, France EDU/DCD

PISA for Development International Advisory Group Meeting Expected Results from Meeting • Agreement on the main components of the Terms of reference for the International Contractor(s) - General approach Cognitive instruments Contextual questionnaires Next steps for managing the tendering process 2

PISA FOR DEVELOPMENT General Approach

General Approach - 1 • To focus on the tasks as they relate to the development of the cognitive instruments and of the context questionnaires. • The elements relating to other tasks (e. g. sample design, translation and verification, survey operations, analysis and reporting) are not covered in this discussion and will be added in the final To. R. • The elements relating to approach to OOS are also not covered in this discussion: these will be focus of a separate tendering exercise 4

General Approach - 2 Focus on: • the constraints that should be imposed on the bidders for the International Contract(s); • the requirements expected of bidders; and • questions that the bidders should respond to as part of the tendering process 5

General Approach - 3 • principles and components of the To. R would, once agreed by the International Advisory Group (IAG) of the PISA for Development project, provide the framework for the development by OECD of a complete To. R and associated tendering documents that would serve as the basis for a call to tender by the OECD 6

General Approach - 4 To. R and tendering process will: • reflect the general PISA principles; • reflect and support the rationale of the countries for participating in PISA for Development; • emphasise capacity building; • emphasise peer-to-peer learning • emphasise consultation and communication with and support for participating countries • emphasise track record of bidders in building capacity for student assessment in developing countries 7

PISA FOR DEVELOPMENT Cognitive Instruments

To. R – Cognitive Instruments Countries require an assessment that: • Reports results on the PISA scale and evidence supports comparability to international PISA results; • Allows students to demonstrate the full range of proficiency levels; and • Adheres to all PISA standards – unless certain modifications are agreed for the implementation of PISA for Development. 9

To. R – Cognitive Instruments Fundamental constraints for bidders: • No new cognitive items will be developed • Review the secure pool to inform the selection of items based upon their cultural and contextual suitability • The targeted test should give a robust and targeted measure of the country's strengths and weaknesses 10

To. R – Cognitive Instruments Fundamental requirements of bidders: • Assess students' abilities as they are now, note where it is thought the students’ performance should be. • Better targeting but also assess across the full range of the PISA scale. • Must ensure sufficient PISA framework coverage and coverage of a full range of proficiency levels. 11

4 Key elements of the To. R and bidding documents • Review of assessment framework and items – including cross-cultural validity and test targeting; • Test design; • Review of the proficiency levels; • Review of scaling models. 12

Review of assessment framework and items - 1 Constraints for bidders: • The selection of all assessment items is based on the PISA assessment frameworks for reading, mathematics and science • Any extension of the framework will need to continue to incorporate the original 13

Review of assessment framework and items - 2 Requirements of bidders: • To carry out a complete review of the assessment frameworks • Consider the implications of PISA 2015 move to a computer-based assessment 14

Review of assessment framework and items - 3 Requirements of bidders: • Need to review PISA’s technical standard on language of instruction? • Processes to review item suitability, translation and verification of test – Variations needed? 15

Review of assessment framework and items – 4. Cross-cultural validity • Item-by-country interactions (country DIF). • Need to examine the potential impact of these interactions on the validity of reporting. • Bidders’ approach to this and proposed solutions. 16

Review of assessment framework and items – 5. Test targeting • Cognitive instruments to be developed from the pool of 337 secure PISA items • Bidders’ proposals for test design should: – deal with the challenge of designing a test that draws upon easier items – But still matches the framework specifications, including with regard to item-types 17

Test design - 1 Requirements of bidders: • To reliably forecast from a targeted test the proportion of participating students who would successfully complete the most difficult items • Achieves the desired test targeting but – provides good coverage of the frameworks – provides results comparable with the main PISA assessment • Propose an equating study 18

Target population • International target population: – 15 -year-old students attending educational institutions in grades 7 and higher • Bidders to make provisions to allow a participating country to supplement the sample 19

Review of the proficiency levels • Propose how to ensure these are meaningful • How to better describe the proficiency of students who perform below the lowest proficiency levels 20

Review of scaling models - 1 Should PISA for Development pursue alternative scaling models that allow features such as: • varying discrimination across items; • dependencies between items clustered in units • guessing; and, • parameter variation (including difficulty) across countries? 21

Review of scaling models - 2 • To what extent do the observed item-bycountry interactions undermine developing a PISA-like learning metric for developing countries? . • How can their proposals for scaling overcome this? 22

Draft Components of the To. R for the IC(s): Contextual Questionnaires Tramonte & Willms Paris, May 28, 2014

The 7 themes • early learning opportunities • language at home and school • family and community support • quality of instruction • (effective) learning time • student socioeconomic status (SES) • school resources • Add items on students’ early learning experiences, • Add items on students’ familiarity with the language of the test • Measure parental involvement, social capital and cultural capital • Measure the role of other community members and of types of community • Enhance measure of school attendance • Add items on participation in formal and informal labor market

The last two themes – student SES and school resources – should be emphasised in the To. R for the International Contractor(s). In presenting their proposals, bidders will be asked to consider the guidance provided by the expert paper, to elaborate extensively on the last two themes, and to justify alternative approaches.

Implications for the Components of the To. R for the International Contractor(s) The 7 themes will be addressed with theoretical, methodological, and technical discussions in the To. R for the International Contractor(s).

Requirement in the enhancement of the questionnaires according to the 7 themes The revised questionnaires must be: a) comparable internationally and b) consistent with the current PISA frameworks.

Structure of the To. R-contextual questionnaires This component of the To. R is articulated in three parts: • Part One refers to the first 5 themes that require minor intervention; • part Two and Three refer to the two critical themes, SES and school resources, that warrant greater attention; • finally, the contextual component of the To. R contains indications on general expectations, specific questions, and underlying considerations on choice of informants and synergy with national assessments.

Key elements of the To. R: Part One Five areas for enhancement and measurement: üearly learning opportunities, language at home and school, family and community support, quality of instruction, learning time. Four underlying aspects for bidders to respond to: üthe best informants; üthe core questionnaires; üPf. D and its synergy with national assessments; üthe collection of data on quality of instruction that are subject specific;

Key elements of the To. R: Part Two One area for enhancement and measurement: üStudent SES Four underlying aspects for bidders to elaborate on: ü offer options to address the extension of SES: adding to PISA ESCS more items at its lower end and/or develop new “poverty-related” measures; ü capture the experiences of different countries regarding their own variables for measuring SES; ü create a global measure of poverty that can be applied consistently across countries for comparative purposes; ü ensure comparability of results of the project on ESCS scale with international results, even if the scale is extended.

Key elements of the To. R: Part Two Requirement for bidders: the measure of socioeconomic status (SES) that is developed must be comparable across participating countries and with the results of PISA international assessment, even if the scales are extended. In addition, a measure of SES for Pf. D should be: ü a reliable and valid measurement of SES within each country; ü a tool for accurate assessment of low levels of SES and poverty; within each country and across countries; and ü a comparable measure of SES and its variability across countries.

Key elements of the To. R: Part Three Area for enhancement and measurement üMeasuring school resources OECD and participating countries agreed on the need to integrate the PISA questions with few new measures on school resources that provided data on basic services, didactic facilities, and didactic materials. The bidders should enhance the PISA tools to capture: üavailability, conditions, and use of basic services, didactic facilities, and didactic materials; üconditions and quality of school infrastructure; üschool infrastructural features, services, and safety.

Key elements of the To. R: Part Three Expectations and challenges: üBidders should propose viable options for maintaining comparability with the main PISA instruments where essentially the same construct is being measured. They should therefore explain the extent to which their proposal will achieve this.

ü Bidders should explain how to capitalise on potential synergies in data collection with ongoing national/international assessments and üthey should identify who are the optimal respondents about availability, conditions, and use of basic services, didactic facilities, and didactic materials. ²Note: this issue relates to the broader discussion on the extension of PISA core questionnaires to accommodate teacher or parent questionnaires (see later).

General Expectations: Bidders should provide a detailed account of: üthe methods proposed to measures of SES and school resources, as well as to conduct pilot testing for the revised questionnaires; üthe strategies to deal with the challenges associated with pilot testing, obtaining reliable and valid data from school administrators, and integrating the new content into the current PISA framework.

General Expectations (continued) Bidders should provide a detailed account of how to: üensure international comparability once new measures are introduced; ümaintain comparability with PISA current measures; üallow nationally relevant analyses; üprovide data that inform of equity and equality.

Expanding on questionnaires and informants. Three questions for the bidders Currently PISA assesses students and schools with two core questionnaires (student and school); parent and teacher questionnaires are optional. 1. Given the 7 themes that need to be captured in Pf. D, are the core questionnaires enough? 2. If new questionnaires have to be introduced, what should they be? Why? 3. Should any of the core questionnaires be dropped? Why?

More specific questions: üWho are the best informants for the 7 themes? üHow can bidders insure that the informants provide effectively their information? üHow do bidders ensure that the current PISA questions and the new questions are relevant for the partner countries? üHow do bidders plan on collecting data on quality of instruction that are subject-specific? üHow do bidders propose to capture students’ learning time, in and out of school?

On Informants The bidders should discuss the rationale for choosing their informants and the need for expanding the set of core questionnaire to guarantee: ülimited number of non-responses or missing data resulting from: ²low level of literacy of the respondents; ²impoverished living conditions of the respondents ²gap between language of the informant and language of the questionnaire. ürepresentation of the variability in family structures; ücomparability between and within participating countries; ütriangulation of information from different questionnaires; ücontained length of the questionnaires

Pf. D and National/International Assessments Pf. D data should be a tool and a resource for each participating country. Bidders should propose how to maximize the synergy between Pf. D and national assessments. While maintaining the core PISA content, they should explain how to ensure: üstability and consistency of data collection within the country; ücomparability of data between Pf. D countries and PISA countries; üvalidity and reliability of the indices used to conduct meaningful analyses within each country; and ümanageable national datasets that can be analyzed by Pf. D national analytical teams.

On sampling and methods Bidders are expected to design national samples that are structured using the same stratification variables used in PISA but also incorporate country specific variables. In the bid, applicants are expected to propose innovative methods of data collection and data merging that allow for effectively linking Pf. D data to locally-collected national datasets.

CALL FOR TENDER PROCESS Roles, Responsibilities and the Process

Call for Tender process Set clear evaluation criteria Agree structure of Cf. T Draft Terms of Reference Independent experts; Secretariat; IAG; SDG potential bidders Deadline for Submission Technical Review Panel Finalisation Identify potential bidders Call for Tender issued

Technical Review Process (detail) • • The technical criteria reviewed prior to the panel meeting to ensure common understanding among panel members. Panel members individually score bids on the technical criteria (without knowledge of the budget proposals) prior to convening for the panel meeting. The panel convene in person to share scores and to discuss reasons for score deductions. Cost proposals reviewed after all the bids are discussed. Follow-up questions sent to the top-rated bidders. The Secretariat drafts the report, incorporating responses to questions, for the panel’s review. The panel agrees the report and the recommendations therein.

Call for Tender process Agree structure of Cf. T Terms of Reference Deadline for Submission Technical Review Panel Draft Finalisation Technical Review Report Call for Tender issued IAG/SDG review Contract negotiations Draft contract Procurement Board Contract signed

Best Value for Money • OECD awards the tender based on the concept of best value for money – Best quality to price ratio – Bidders offering added value can improve the quality of the goods and services purchased by OECD – Adds a level of complexity which makes the Tender Evaluation Process important – Evaluation Criteria balances • • • Contract compliance Performance Delivery Ability to meet minimum specifications Price

Timeline and procedures: 2014 Agree structure of Cf. T Draft Terms of Reference • NJune • No. June • August • Dec. July • September IAG/SDG review • 7 September • 30 -31 May 2011 Draft contract • 13 October 2011 • 1 July Technical Review Report Technical Review Panel Deadline for Submission Call for Tender Finalisation Procurement Board • 2 October • 30 -31 May 2011 • 10 October Contract signed • 7 November

Development Capacity Needs Assessment Framework Fernando Cartwright

Objective Develop a framework to identify any factors or conditions that are barriers to the successful implementation of PISA, including the production of meaningful information and consumption of information by stakeholders, in participating countries.

Needs Assessment Dimensions · Enabling environment · · relationships, Organization · · Legislative, political/bureaucratic cooperating partners the National Centre and other sub-national institutions Individual · National Centre staff, data collection teams, local service providers

Program goals and Project Management · Intended Program outputs · · Project requirements and activities · · Defined by PISA scheduled activities Quality standards · · Macro-level goals of PISA for Development PISA technical standards, SABER – Student Assessment Specific project elements · Project management tasks required for successful implementation Sources: PISA Technical Standards, NPM Manual, PISA for

Systems Approach for Better Education Results (SABER) – Student Assessment

SABER rubrics Development Level Dimension ESTABLISH LATENT EMERGING ED ADVANCE (Absence of, (On way to (Acceptabl D or deviation meeting e (Best from, minimum practice) attribute) standard) Justificati on

Enabling Environment ratings: normative definitions 1. Latent: there is no environmental support or there are environmental obstacles that deter program implementation. 2. Emerging: there are political, economic or social structures in place that may be adapted to facilitate implementation. 3. Established: political, social or economic structures exist that can support implementation. MINIMUM REQUIRED 4. Advanced: political, social or economic structures are currently providing support to similar activities.

Organizational ratings: normative definitions 1. Latent: there is no capacity to assume this role. 2. Emerging : some capacity exists but it is not institutionalized in a coherent administrative structure. 3. Established: some capacity exists within a coherent administrative structure, but may lack availability or technical skills to assume responsibilities. MINIMUM REQUIRED 4. Advanced : capacity is institutionalized and has sufficient resources to assume the responsibilities without developing additional capacity.

Individual ratings: normative definitions 1. Individuals do not have the skills and/or are resistant to developing requisite skills 2. Individuals have foundational knowledge or personal attributes that will enable them to acquire requisite skills or attributes 3. Individuals have sufficient knowledge, interest and aptitude to allow development of requisite skills or attributes with brief workplace training and/or independent training and practice. MINIMUM REQUIRED 4. Individuals already have the required skills or attributes

Operationalization

PISA Element / Description Activity Latent Adequacy of Facilities of transportatio the National no vehicles n for data Centre collectors Recruitment Commitment Insufficient and training of data collection of test collection administrator staff s Recruitment and training Availability of training of test administrator facilities s Recruitment Commitment and training of data of test collectors to administrator training s No facilities available (selfstudy or oneone-one) Emerging Establishe Advanced d public/shared personal transportation vehicles part-time shared with other institutions institutional dedicated vehicles part time with specially hired same for this institution role/project Existing facilities may be repurposed to accommodate training Dedicated training environment is available Training time is compensated Data collectors Data collector and is must volunteer time is integrated with time or training compensated regular duties conflicts with but (or staff are other responsibilities hired

Capacity Building Plan

Prioritization and Trade-offs 1. Meet minimum standards for capacity 2. Prioritize immediate requirements within the PISA cycle 3. Added-value outcomes associated with PISA implementation 4. ‘Wish-list’ items

PISA Element / Description Activity Latent Adequacy of Facilities of transportatio the National no vehicles n for data Centre collectors Recruitment Commitment Insufficient and training of data collection of test collection administrator staff s Recruitment and training Availability of training of test administrator facilities s Recruitment Commitment and training of data of test collectors to administrator training s No facilities available (selfstudy or oneone-one) Emerging Establishe Advanced d public/shared personal transportation vehicles part-time shared with other institutions institutional dedicated vehicles part time with specially hired same for this institution role/project Existing facilities may be repurposed to accommodate training Dedicated training environment is available Training time is compensated Data collectors Data collector and is must volunteer time is integrated with time or training compensated regular duties conflicts with but (or staff are other responsibilities hired

PISA Element / Description Activity Latent Adequacy of Facilities of transportatio the National no vehicles n for data Centre collectors Recruitment Commitment Insufficient and training of data collection of test collection administrator staff s Recruitment and training Availability of training of test administrator facilities s Recruitment Commitment and training of data of test collectors to administrator training s No facilities available (selfstudy or oneone-one) Emerging Establishe Advanced d public/shared personal transportation vehicles part-time shared with other institutions institutional dedicated vehicles part time with specially hired same for this institution role/project Existing facilities may be repurposed to accommodate training Dedicated training environment is available Training time is compensated Data collectors Data collector and is must volunteer time is integrated with time or training compensated regular duties conflicts with but (or staff are other responsibilities hired

PISA Element / Description Activity Latent Adequacy of Facilities of transportatio the National no vehicles n for data Centre collectors Recruitment Commitment Insufficient and training of data collection of test collection administrator staff s Recruitment and training Availability of training of test administrator facilities s Recruitment Commitment and training of data of test collectors to administrator training s No facilities available (selfstudy or oneone-one) Emerging Establishe Advanced d public/shared personal transportation vehicles part-time shared with other institutions institutional dedicated vehicles part time with specially hired same for this institution role/project Existing facilities may be repurposed to accommodate training Dedicated training environment is available Training time is compensated Data collectors Data collector and is must volunteer time is integrated with time or training compensated regular duties conflicts with but (or staff are other responsibilities hired

Capacity Building Tools 1. Organization of existing resources 2. Definition of protocols to increase bureaucratic efficiency 3. Local expertise and service providers 4. International technical assistance and training 5. PISA International Contractor

Some issues for Capacity Building Plan Funding agreements and project budgeting Infrastructure vs human resources Mo. E internal capacity vs. service providers Short-term PISA capacity vs long-term large-scale assessment programming

PISA for Development 1 st International Advisory Group Meeting To. R for an independent project review 27 – 28 May 2014 Paris, France EDU/DCD

PISA for Development: Independent review Scope and purpose of the review • focused on the progress of the project in relation to its five main outputs and extent to which the delivery of these will achieve the project’s purpose • purpose of review is to help understand what has been achieved against the OECD’s original plans, how practicable those plans were as well as how relevant and valuable the project’s work is to developing countries’ and development partners' evolving education policies. 67

Project’s outputs – Contextual questionnaires and data-collection instruments enhanced; – Descriptive power of cognitive assessments enhanced – Analytical framework and methodological approach for including out-of-school 15 -year-olds in assessments developed; – Country capacity in assessment and analysis strengthened – Engagement established with developing countries and partners for peer-to-peer analysis and learning opportunities to support the UN-led post-2015 process 68

Main issues to be reviewed • Impact of the project • Relevance of the project • Sustainability of project achievements • Management and partnership arrangements 69

Impact of the project • Achievement of outputs and the purpose of the project • Expected impact on policy makers and the education systems in the participating countries • Evidence and indications that PISA results will be used in policy-making • Dissemination and use of project deliverables 70

Relevance of the project • Key lessons from the project to inform work on improving education quality and improved student learning outcomes • The role of the project in informing discussions of education quality and learning outcomes • Promotion of evidence-based policy making 71

Sustainability of project achievements • Achievement of capacity building outputs and objectives • Sustainability of the capacity that has been built • Lessons from capacity building • Success of peer-to-peer learning strategies • Likely transition of countries from PISA for Development to main PISA • Spill-over benefits of the project for student assessment as a whole 72

Management and partnership arrangements • Effectiveness of the governance and management structures for the project • Effectiveness of project management systems and processes • Roles of PISA GB, the DAC and IAG and TAG • Particular successes and challenges in implementing the project 73

Methodology • Selection of experts • Experts to propose a design, plan and methodology in accordance with To. R • Collection and analysis of documents, data and information, interviews with stakeholders and review of documents produced by the project • Possible use of surveys 74

Schedule Inception Report by December 2016 Presentation of initial findings and End February 2017 recommendations Draft Report Mid-March 2017 Presentation to IAG Late March 2017 Final Report May 2017 75

Deliverables • Final report of 50 pages • IAG and TAG will have opportunity to comment on Inception report and Draft report • Final report will inform OECD’s final report on the results of the project 76