Pacific Islands Literacy and Numeracy Assessment PILNA 2012

Pacific Islands Literacy and Numeracy Assessment (PILNA) 2012 Nina Tu’i OCIES Conference Panel 4 -6 November 2015 Port Vila, Vanuatu

Context • International Large Scale Assessments (ILSAs) – Education policy informed by evidence • Regional or National level LSAs – Capacity building and community engagement • Sustainable development goals – Monitoring – Targeted interventions

Research methodology • Conducted interviews with all those who were involved at the regional level of implementation • Questionnaire for national level coordinators of PILNA 2012

Background • 14 PICTs: Fiji, Federated States of Micronesia, Kiribati, Solomon Islands, Tuvalu, Papua New Guinea, Samoa, Cook Islands, Nauru, Tokelau, Palau, Niue, Republic of the Marshall Islands and Vanuatu • More than 25, 000 Year 4 and 6 students assessed for literacy and numeracy

Purpose • Call from Pacific Education Ministers for more data on Literacy and Numeracy in PICTs • Establish a baseline of literacy and numeracy levels of year 4 and 6 Pacific Island students • Well understood by those implementing at the regional level

Processes • Design – Jointly between SPBEA and UNESCO-Apia – Sampling • Implementation – Instrumentation – Delivery of Assessment – Processing • Reporting

Use of PILNA 2012 results 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Read a report Heard a presentation of PILNA results There was an There was a discussion of internal media release PILNA results communication about PILNA amongst officers within the results Ministry about PILNA results % of total responses Heard casual conversations about PILNA results Other

100% Dissemination of PILNA 2012 results 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Principals or head teachers Community members (e. g. % of total responses parents) Others

Impact of PILNA 2012 results 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Plans Interventions % Yes % No New Initiatives

Lessons learnt from PILNA 2012 • • • Clarity of purpose Capacity development Sampling Strategy for use of results by countries Design of assessment incorporating processing

2015 PILNA Torika Taoi OCIES Conference Vanuatu

2015 DESIGN The design seeks to address the two objectives: i) to develop a tool and processes that are grown out locally from within the region for delivering long term valid and reliable regional assessment data ii) to lift regional capacity in collecting and working with regional assessment data.

Country agreement November 2014: invite sent to countries Conditions for participation: The two key conditions require that: • participating countries are committed to sharing the results with other countries for lessons that one can learn from each other. • each country is committed to using the findings to carry out policy interventions as well as technical interventions aimed at improving the situation in each country.

PILNA 2015 and FEd. MM Results of 2012 were presented to 2014 FEd. MM In response to these findings: • FEd. MM endorsed re-run of PILNA in 2015 for all PICs countries but based on funding availability. • FEd. MM also called on Development Partners to work with countries to identify courses of poor performance. • Focus on identifying specific areas of weaknesses regionally and at country level; using evidence to identify regional and country interventions. • Adopting a benchmarking approach using evidence from 2015 PILNA • More diagnostic

Purpose of PILNA 2015 • Diagnose literacy situation in Pacific to identify main courses of low performance. • To trial questionnaires: Require additional data based on factors influencing student performance such as; - school leadership, teacher quality, student assessment, curriculum - parental support, - socio-economic status of families, - policies in place.

PILNA PROCESS Develop Parallel Items: • review of the 2012 test items, including its psychometric analysis was carried out in December 2014, to allow for comparability of 2015 results with the 2012 results. • Parallel items developed and also new items constructed to fill in gaps identified from the 2012 set od learning outcomes

PILNA Process Components assessed in Literacy • Writing • Reading and Language features Components assessed in Numeracy • Numbers-Recognising Numbers, Place values & Relations, Fractions & % • Operations- Addition, Subtraction, Multiplication, Division(Y 6 only), Problem Solving • Measurement • Data (only for Year 6) The cognitive skills were the same for both Year levels, but the skill demand increased in Year 6. These differences in skill demands reflected the learning outcomes that tapped into the different ability levels of the students

PILNA PROCESS Review of items for trial • Items panel reviewed by EQAP officers before assembling for trial Trial run • As there were new items developed, these had to be trialled. Items were trialled in 5 countries (Pa. BER countries and 2 non-Pa. BER countries): Samoa, SI, PNG, Kiribati and Fiji

PILNA PROCESS Data Scoring • Different from 2012 • 2015 – Degree of correctness : 0, 1, 2, 3, 9 • 2012 – Levelling : L 0, L 1, L 2, L 3, L 4 2012 2015 Scoring and Levelling Scoring: L 0 – Inability to demonstrate proficiency in any of the skills that are basic 0 – 0% correct L 1 – Not yet working at expected level 1 – partially correct; - 100% correct L 2 – Working towards at expected level 2 – partially correct; - 100% correct L 3 - Performing at expected level 3 – 100% correct L 4 – Performing above expected level 9 – no answer

PILNA PROCESS Data processing and Data analysis • Samoa, Solomon Islands, PNG, Fiji (Kiribati) – In-country and central scoring – Electronic data entry • Cleaning – EQAP + Country (NCs) • IRT analysis Assemble items for main test: • Items were finally reviewed • Spelling and typo errors edited

PILNA PROCESS Translation • In consideration of Language Policies or languages of instructions applicable at lower primary in the PILNA participating countries. • PILNA instruments are translated into 8 country languages (Cooks, Niue, Tonga, Vanuatu, RMI, Tokelau, Kiribati & Tuvalu • Only 4 countries opted for translated versions in 2012

PILNA PROCESS REPORTING • Data will be analysed and reports will be developed in the first quarter of 2016. As with PILNA 2012, it is anticipated that reports will be developed at the following levels: • Country • Regional • Sub-regional levels

Sampling The sampling design is complex, given the extreme variations that apply across the countries participating in PILNA. • Census Approach- applied to relatively smaller countries: o Cook Islands, Niue, Palau, RMI, Tokelau, and Tuvalu (but with certain schools excluded). • Sampling approach- applied to FSM, Kiribati, PNG, Samoa, Solomon Islands, Tonga and Vanuatu (also with certain exclusions). The sampling arrangements for Papua New Guinea, Samoa and Solomon Islands reflect decisions made through the Pa. BER processes. Fiji and Nauru are not participating in the PILNA 2015.

Sampling • In the sampling frame, schools are categorised by: country, education district/province/region, location address, locality (urban, non-urban, remote, very remote) • school authority (government, private, religious authority) • regional authority (government, non-government) • school size, Year 4 size, and Year 6 size.

The Pacific Islands Literacy and Numeracy Assessment (PILNA) – Strengthening Regional Educational Assessment ‘Uhila-moe-langi Fasi OCIES Conference Port Vila, Vanuatu 2015

SURVEY INSTRUMENTS Students factorss Students performance School environment Home environment

language Study habits Student factors attitude

Where student lives House type Socio-economic factors Home Environment Number of meals Home possessions

Qualifications Pedagogy Experience School Environment: TEACHERS Professional development Community involvement Assessment knowledge and skills

Qualifications and training Management School Environment: HEAD TEACHER Leadership skills and experience Staff development

Research Questions: How do the socio-economic factors influence student performance in PILNA? How does the student study habit influence his/her achievement in PILNA? What influence does teacher quality have on student achievement in PILNA? How does school management and leadership influence student achievement in PILNA?

Instrument: Questionnaires STUDENTS QUESTIONNAIRE TEACHERS QUESTIONNAIRE HEAD TEACHERS QUESTIONNAIRE

Instrument development process: Draft – EQAP Review – EQAP Expert review – ACER Trial Implement

The Pacific Islands Literacy and Numeracy Assessment (PILNA) – Strengthening Regional Educational Assessment Elizabeth Cassity OCIES Conference 2015

A framework for assessment Educational decision making loop Feedback / Evaluation Understanding of current situation ACTION Knowledge about how to improve Required resources Improved outcomes Improved life consequences Masters, 2013

How is the analysis being conducted for PILNA 2015 (in collaboration with EQAP)? Psychometric support – Item Response Theory model • Sampling support – strategy that states sampling objectives, accounts for diverse populations and other parameters • • Questionnaire support – feedback on 2015 pilot questionnaires; support for future cycles • Support for trend measures – providing advice on linking PILNA across years

What are plans for reporting and dissemination for PILNA 2015? • Plans outlined at PILNA Steering Committee Meeting, August 2015 - focus on student learning outcomes is paramount • Six reporting and dissemination priorities: 1. Interventions 2. Policy 3. Political support 4. Community awareness 5. Monitoring results 6. National validation

What is the long term plan for PILNA? • Regional assessment conducted every three years • Linking of PILNA cycles to measure trends • Questionnaires distributed to all teachers, head teachers and students participating in PILNA (post-2015) • Ongoing development of specification and