External and Internal School Evaluation a New Zealand

External and Internal School Evaluation: a New Zealand perspective Presentation to School Inspectorate Training days Bilbao, Basque Country June 2018 Sandra Collins Senior Education Evaluator Evaluation Services Education Review Office

Overview of my presentation 1. New Zealand schools 2. ERO in the New Zealand education system 3. OECD review – a review of ERO’s evaluation indicators 4. ERO’s approach to external evaluation in schools 5. Building internal evaluation capability 6. Balancing internal and external evaluation 7. Evaluator training and development 8. Evaluator capabilities 9. Principles of practice

1. New Zealand schools - overview • Approximately 2400 schools • Most are state schools and government funded • Self managing with high level of autonomy • Most schools have between 100 and 2000 students Our biggest challenges – disparity and within school variability Ko te Tamaiti te Pūtake o te The Child – the Heart of the Matter Kaupapa

2. The New Zealand education system and the Education Review Office (ERO) Parliament Tertiary Education Commission Ministry of Education • • Policy Funding New Zealand Qualifications Authority Minister of Education Review Office Schools and Early Learning Services Parents and the community Education NZ Education Council

ERO’s mandate • An independent government department established in 1989 • Statutory Powers of the Chief Review Officer (s 325 -328 Education Act, 1989) • Right to enter schools and early childhood services • Evaluate the performance of schools and early learning services and publicly report our findings • Designated Review Officers collect authentic, triangulated information in schools and services and verify policy implementation and quality

ERO – who we are and what we do • 142 designated review staff • 80% of staff are field reviewers • National office and 7 regional offices • 650 -840 school evaluations per year • 1200 -1460 early childhood evaluations • Up to 20 national evaluation reports produced each year

ERO’s dual purpose Accountability Improvement Ko te Tamaiti te Pūtake o te The Child – the Heart of the Matter Kaupapa

3. Reviewing ERO’s school evaluation indicators OECD review- trigger for change Ko te Tamaiti te Pūtake o te The Child – the Heart of the Matter Kaupapa

School evaluation indicators conceptual framework

Outcome indicators “confident, connected, actively involved, lifelong learners” Confident in their identity, language and culture as citizens of Aotearoa New Zealand Socially and emotionally competent, resilient and optimistic about the future A successful lifelong learner Participates and contributes confidently in a range of contexts - cultural, local, national and global

Domains - what matters most Process indicators • stewardship of the school • leadership of conditions for equity and excellence • educationally powerful connections and relationships • responsive curriculum, effective teaching and opportunity to learn • professional capability and collective capacity • improvement and innovation through evaluation, inquiry and knowledge building http: //www. ero. govt. nz/publications/school-evaluation-indicators/indicatorsbackground-papers/

Domain 4: Responsive curriculum, effective teaching and opportunity to learn (an example) Evaluation indicator Effective practice Students have effective, sufficient and equitable opportunities to learn. The learning environment is managed in ways that support participation, engagement and agency in learning. Learning opportunities enable students to relate new information to prior knowledge and to modify existing conceptions as necessary. Students experience an environment in which it is safe to take risks and errors are regarded as opportunities for learning. Participation in effective heterogeneous (mixed ability) group activities provides students with cognitive challenge and opportunities for deep learning.

ERO’s external evaluation focus 4. ERO’s current approach to school external evaluation Learner outcomes The conditions that contribute to learner success

ERO’s external evaluation will • Use a school’s evidence – from a range of sources and evaluation activities (triangulate) • Consider and seek further evidence when necessary • Analyse to unpack what the evidence shows- what is so? • Synthesise to decide what the evidence means – so what? • Develop emerging evaluation findings and discuss them with the school governance (board of trustees) • Write a report based on the evaluation judgements • Confirm the report with the board of trustees • Publish the confirmed report on the ERO website

ERO’s evaluation also includes a focus on compliance with legislation Board Assurance Statement (BAS) (can be downloaded from the ERO website) The board chairperson and principal attest to compliance with legislative requirements. ERO checks the BAS as part of preparing for the external evaluation. There is an opportunity for ERO to discuss or clarify anything the board of trustees is not clear about.

Compliance - student safety

ERO’s evaluation process Reporting up Notification • Set School is due for an external evaluation based on previous evaluation • Email notification • 6 -8 weeks before external evaluation • Time allocated to evaluation – depends on size of school and number of evaluators in team • Initial documentation • Previous report Scoping Evaluation and reporting design history • Any complaints on file • System-level achievement and engagement data • Data Onsite gathering • Meetings with key personnel • Document analysis • Observations • Analysis • What is so? • Synthesis • So what? • Emerging findings • Draft report written • Decision about timing of next review made for inclusion in the report • Quality assurance • Peer review • Management review • Draft report sent to school • Response from school • Report confirmed and published on ERO’s website • Post review questionnaire sent to schools

Deciding the timing of the next review http: //www. ero. govt. nz/assets/Uploads/Decision-Next-ERO-Evaluation-SCHOOLS-January-2018. pdf

Timing of next review -2017 data ERO Review Status at July 2017 (proportion of students) ERO Review Status at July 2017 (proportion of schools) 4 -5 year 21% Other 1% 1 -2 year 8% 4 -5 year 35% 1 -2 year 4% Other 1% 761, 000 students 2, 400 Schools 3 year 60% 3 year 70%

System-level evaluations National evaluation reports (NETs)

5. Building internal evaluation capability • School Evaluation Indicators: Effective Practice for Improvement and Learner Success • Effective School Evaluation How to do and use internal evaluation for improvement • Effective School Evaluation for Improvement Good Practice Report http: //www. ero. govt. nz/videos/ www. ero. govt. nz

Domain 6: Evaluation, inquiry and knowledge building for improvement and innovation Indicators Coherent organisational conditions promote evaluation, inquiry and knowledge building. Collective capacity to do and use evaluation, inquiry and knowledge building sustains improvement and innovation. Evaluation, inquiry and knowledge building capability facilitates engagement with external evaluation and the wider education community.

6. Balancing internal and external evaluation An evaluation approach that balances ERO’s external evaluation with internal evaluation, according to each school/service’s circumstances. ERO building schools’ internal evaluation capability ERO evaluation external School evaluation internal Schools experiencing difficulties with very limited internal evaluation ERO return time 1 -2 year Schools operating well with established processes of internal evaluation 3 year School using high quality self review information as basis for decision making Schools sustaining high quality performance and continuous improvement through effective internal evaluation 4 -5 year

School – internal focus ERO –external focus What are the outcomes that are valued for all learners in this school community, as learners in Aotearoa New Zealand, and as global citizens? How well does the school achieve equitable and excellent outcomes for all learners? How well are all our learners achieving in relation to those outcomes? To what extent is every student in our school a successful ‘confident, connected, actively involved lifelong learner’? How effectively does this school respond to those Māori and other learners whose learning and achievement need acceleration? How well are we identifying and accelerating the achievement of those learners at risk of not achieving equitable outcomes? What school processes and practices are effective in enabling achievement of equity and excellence? How do we know? What sources of evidence tell us about our performance and effectiveness?

School preparation for external evaluation Areas of focus improving outcomes Effectiveness of actions taken Success and challenges Impact

A purposeful interaction Internal evaluation deepen the scope of external evaluation give a context to the external evaluation provide important insights improve the interpretation of external evaluation findings increase the use of external evaluation findings External evaluation stimulate internal evaluation expand the scope of internal evaluation validate the results of internal evaluation provide an additional perspective include a capacity building role as part of the evaluation

Impact and quality of ERO’s school external evaluation • In the 2016/17 year, 86 percent of respondents to our surveys indicated that ERO’s school evaluations had made, or were likely to make, a contribution to their decisions about how to improve learner outcomes. The responses received often included comments which show a strong appreciation of ERO’s work and an acknowledgement of its positive impact. • ERO uses a national moderation panel to determine the levels of compliance against internal standard procedures. Moderation findings are followed up in quality assurance guidelines and processes. At 96 percent compliance, ERO’s moderation result was within the target range for 2016/17. Source: http: //www. ero. govt. nz/publications/annual-report-201617/

7. Evaluator training and development

8. Evaluator capabilities The capabilities are intended to: • guide professionalisation of our evaluation thinking and practice • guide provision of our evaluation training and professional development • assist individual evaluators to self-evaluate their own knowledge, skills and practice; make decisions about and monitor their own development • underpin quality assurance, both internally and externally • underpin the credibility and integrity of ERO’s evaluation work • support high standards and consistency of our professional evaluation practice across the organisation • be an explicit basis for Individual Development and Performance Management (IDPM) processes.

9. Principles of professional practice • A focus on the learner and the equity and excellence of education outcomes. • Promoting external accountability and strengthening internal accountability. • A culturally and contextually responsive approach. • The integration of internal and external evaluation. • Technical rigour in evaluation design; data gathering; analysis, synthesis and reasoning; and the communication of evaluation information. • A participatory evaluation process. • The promotion of evaluation use. • Developing evaluation capacity. • Capable and ethical evaluation practice.

The integration of internal and external evaluation (an example of one principle of practice) This means that we will • use the education institution’s internal evaluation of how well it is achieving equity and excellence of outcomes for learners to focus our external evaluation discussions and activities • start by considering what the school knows from its internal evaluation about its own effectiveness in achieving equitable and excellent outcomes for learners and how well its organisational conditions support or inhibit improvement • discuss with participants their interpretation and evaluative thinking about their progress and achievement data and other information • evaluate the quality of internal evaluation processes and test the validity of internal evaluation findings and the effectiveness of internal evaluation processes • evaluate the effectiveness of internal evaluation processes and improvement actions in ensuring responsive education provision and equitable opportunity to learn for every young person and the school’s capacity for sustained improvement • use ERO’s evaluation resources to promote the integration of internal and external evaluation • enable schools to purposefully engage with external evaluation and use it as an opportunity to review, develop and validate its internal evaluation and identified improvement actions.

School evaluation has both internal and external components, and the two should be seen as working together. New Zealand has. . . gone furthest among countries internationally towards a collaborative school evaluation model, incorporating at the same time a sequential process. . . [the] approach is collaborative in the sense that both parties attempt to work together to agree on a rounded picture of the school in which there is mutual recognition of its strengths and consensus on areas for development. Nusche, D. , Laveault, D. , Mac. Beath, J. , & Santiago, P. ( 2012). OECD Reviews of Evaluation and Assessment in Education: New Zealand 2011, OECD Publishing (p. 105).

Internal vs External Evaluation Everybody seems to hate external evaluation while nobody trusts internal evaluation. David Nevo (1995)

FOR MORE INFORMATION sandra. collins@ero. govt. nz www. ero. govt. nz https: //twitter. com/Education. Rev. NZ