Use of indicators for educational outcome by the

Use of indicators for educational outcome by the Dutch Inspectorate. Pitfalls and new challenges in a network society SICI conference ‘Assessment of Educational Results Prague 2830 march 2017 Bruno Vreeburg Dutch Inspectorate of Education

Four questions to start 1. Is the use of indicators for educational outcome for accountability of schools by an inspectorate always necessary? When not? 2. Do you deal with undesirable strategic behavior in schools when the inspectorate uses the assessment of educational results for accountability? How to prevent? 3. We think norms and strong benchmarks for the minimal level of schools are necessary. But how about the better schools? Do the inspectorates need a differentiated use of educational results for accountability? How to design? 4. Can schools give their own calculation of educational results for accountability? When it is valid? How to compare? 5. …. . 6. …. .

Contents of the presentation 1. 2. 3. 4. 5. 6. 7. 3 Educational information in the Netherlands Indicators for schools about educational outcome Assessment information for accountability and strategic behaviour Information at the system level Discussion about outcome indicators Development of outcome indicators Final remarks

THE EDUCATIONAL SYSTEM IN THE NETHERLANDS AND THE ASSESSMENT INFORMATION 4

Data DATA Compulsory Standardized Final Test 12 year - Primary Education - Special Needs Education Intermediary test Core subjects Diagnostics Final Examination - Standardized Central tests (all Subjects) - Schooltests (all Subjects) No central examination in senior vocational education and higher education Senior Vocational Education PSE Primary Education Secondary Education diploma Higher Education Content and remarks 55 Test about core subjects Different test providers Schooladvice for secondary education Not for use by inspectorate Not diploma Examination scores for all the students 5 tracks diploma Only information about educational succes: percentage diplomas, drop-outs, switch, levels and study duration

INDICATORS FOR SCHOOLS ABOUT EDUCATIONAL OUTCOME 6

Three essential conditions for using educational assessment for accountability 1. Indicators of educational outcome at a school level must cover the full width of the results. 2. The indicators for educational outcomes must take into account the relevant circumstances of the school. - Gross and net effects: Correction for student characteristics and value added. 3. The indicators have to be transparant, verifiable (you can recalculate them) and you can influence them. De Bruin (2001), Hargreaves and Braun (2014) 7

Functions of the educational system Three functions: • Qualification: diploma, ticket for labour market • Socialization: citizenship, participation in the society • Allocation: sufficient people in the relevant jobs. Talent development for every child. We can make indicators for the system and for schools. 8

Outcome indicators: school level and system level School System Qualification Learning performances (tests), diplomas, attained level, no drop out x (gross and net) x (gross) Distribution of student performance in different groups x x (male/female, ethnicity, social-economic background) Socialization Citizenship, social cohesion. Hard to observe (only qualitative). Allocation Enough qualifications, labour shortage, distribution of educational outcome by groups. x Placement in a succeeding trajectory x Conditional Efficiency, no retardation x (gross and net) 9 X (gross)

Examples of indicators Primary Education • Final test results assessed over 3 years. • The norm depends on the percentage of children with poorly educated parents. Secondary education • Final test results. • Attained level. • Yearly succes and no retardation. • Correction for poorly educated parents, sometimes entrance level. • Balanced scoremodel. 10

Use of the outcome indicators • Schools use it for quality management • Risk detection • Judgment • Diagnostics (there a lot of diagnostic indicators). 11

Challenges now Not only the information and the outcome indicators of the inspectorate are interesting. More important: • School also have information about outcome. Empowerment of schools with instruments to analyse their outcome themselves. • Encouraging schools to set their own ambitious targets. 12

ASSESSMENT INFORMATION FOR ACCOUNTABILITY AND STRATEGIC BEHAVIOUR 13

Strategic behaviour Desired behaviour: • More effort. Higher results. Undesirable behaviour (schools and students) • Teaching only for the test. • Minimalisation of effort. • No admission of weak students. Schools may exclude pupils whom they fear they will bring down the mean test score. • Slide between encouraging, a little help and cheating during test. • Schools may use test questions as practice material. Lazear (2006), Koretz, Hamilton in: Brennan (2006), Fischbacher, Föllmi-Heusi (2013), Borghans, Schils (2015), Muskens, Tholen (2015) https: //www. onderwijsinspectie. nl/documenten/publicaties/2015/11/10/onbedoelde-effecteninspectietoezicht-onderzocht 14

Strategic behaviour: Distribution primary education Near the ceasura you see a slight deviation from normality 15

Questionaire example. Admission of a student: Suppose a new student is registered for the school and you can expect he will be a weak student. Do take into account the consequences for the judgement of outcome by the inspectorate when deciding to allow or not allow this student? On my own On other schools Not at all 79 20 A little 16. 5 30 Strong 4 22 Don’t know/No answer 0. 5 29 16

Mean scores for the school test (SE) and the standardised central test (CE) Until 2011: Schooltest can compensate standardized central test 17

Conclusions from our research • Strategic behaviour exists and you can detect it at the system level but not always at the school level or student level. • In most cases it’s not very problematic and heavy. • But you have to organize some prevention. 18

OUTCOME INFORMATION AT THE SYSTEM LEVEL 19

Information at the system level Information about student assessment can also be used at the level of the educational system: • Cohort flow and efficiency • Results by group • Trends, Variability, Spread Each year: ‘The State of Education’. • Thematic enquiry • Aggregation of surveillance results Next page: Main message in 2016 about growing inequality. 20

21

Conclusion Research at the system level is not only summing up results of the inspections of schools. Research in its own right. Gives different views and other insights you cannot see at the individual schools. Trend towards: • Surveillance at system level • Thematic enquiries 22

DISCUSSION ABOUT INDICATORS FOR EDUCATIONAL OUTCOME 23

Political discussions and complaints of some schools There is a lot of discussion about the use of outcome indicators by the inspectorate: • Inspectorate incites too much testing. • Anti testing movement. You shall not test children. • Using indicators for accountability incites strategic behaviour. • Inspectorate is too strict (but insufficient outcome according to inspectorate is about 6 - 10%). But also …. 24

Press and publicity For good schools the indicators and the norms don’t give a challenge Press uses outcome information in their own way. Choice lists, rankings and league tables. They say mostly: inspectorate is too soft. 25

Example: A sample of winners and losers according to the press (Elseviers Magazine, jan 2017) 26

Value added score in vocational education 2015. You can at most distinguish three groups Schools with much value added Schools with low value added 27

Difference between press and inspectorate Same data but different view. Inspectorate • Correction for fair judgment • 6. 5% insufficient • No lists In the press • League tables from low to high. They are always fond of lists. • Winners and losers • No correction • 13. 5% insufficient Elseviers Magazine, jan 2017 28

Conclusion The stakeholders have conflicting needs and the inspectorate has to deal with them. 29

DEVELOPMENT OF OUTCOME INDICATORS 30

Response to criticism How can you respond to criticism and reduce risks for strategic behaviour? 31

1. Nudging is an answer to undesirable strategic behaviour. Give people a little push (nudge) in the good direction. General examples of nudging: • Default value in electronical form of student loan is maximum or zero. Last possibility gives a significant smaller loan (the same with a creditcard). • To sign your tax form in advance or after. The first possibily gives a better and fairer completion of the tax form. Oreopoulos, Behaviouoral economics of Education (2015; http: //www. behaviouralinsights. co. uk/ 32

Nudging and educational outcome Balanced score model. (With our data this is not easy in primary education). A small change in an indicator can change behaviour. - Possibilities for recalcultation. - Give more weigth to a correction than statistically is necessary. But ……… more important is the interaction with the stakeholders and empowerment of schools. 33

2. Three design principles 1. Interaction Design indicators together with the stakeholders who have interest in them. Ownership of indicators. 2. Variety Variation in the concepts of outcome and definitions. 3. Dynamics Not only measuring the outcome but also the underlying process. De Bruin (2001), Raad voor Maatschappelijke ontwikkeling (2011), Hargreaves en Braun (2014) 34

3. Empowerment of the schools They have their own opinion about educational results. • Focusing on results • Data driven teaching They analyse results systematically with regard to: . • Individual children/students • Groups • The program • Teacher behaviour. Management has a focus on good results. Visscher, Coe (2002); Hamilton, Halverson e. a. (2009); Visscher (2015) 35

FINAL REMARKS 36

Final remarks: evolution From To • Small data sets. • Only inspectorate has information • Inspectorate constructs indicators. • Emphasis on technical refinement • Inspectorate judges the outcome of school. • Inspection of schools • School level. • Large data sets. • Sharing and other stakeholders have their own information • Joint construction with stakeholders. Joint ownership. • Outcome in a broader context. 37 • Empowerment of schools to set their own standards. • Inspection of school boards. • System level gets more emphasis.

Four questions to start 1. Is the use of indicators for educational outcome for accountability of schools by an inspectorate always necessary? When not? 2. Do you deal with undesirable strategic behavior in schools when the inspectorate uses the assessment of educational results for accountability? How to prevent? 3. We think norms and strong benchmarks for the minimal level of schools are necessary. But how about the better schools? Do the inspectorates need a differentiated use of educational results for accountability? How to design? 4. Can schools give their own calculation of educational results for accountability? When it is valid? How to compare? 5. …. . 6. …. .

Thank you for your attention. Do you have questions? 39

Information about Dutch schools (mostly in Dutch) Website of the Dutch Inspectorate: www. onderwijsinspectie. nl Information about schools by the Dutch inspectorate: https: //zoekscholen. onderwijsinspectie. nl/ Website of the organisations of schools in primary and secondary education with school information: www. scholenopdekaart. nl The State of Education in the Netherlands (English) https: //www. destaatvanhetonderwijs. nl/downloaden/englishversion 40