Consider the Evidencedriven decision making for secondary schools

Consider the Evidence-driven decision making for secondary schools A resource to assist schools to review their use of data and other evidence 7 Changing and Evaluating Page 1 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evidence-driven decision making This module is part of a resource about how we use data and other evidence to improve teaching, learning and student achievement Today we are looking at the final stage of this process – changing our practice and evaluating the impact of that change Page 2 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

The evidence-driven decision making cycle Trigger Data indicate a possible issue that could impact on student achievement Speculate A teacher has a hunch about a problem or a possible action Explore Check data and evidence to explore the issue Reflect on what has been learned, how practice will change Evaluate the impact on the intervention Question Clarify the issue and ask a question Assemble Decide what data and evidence might be useful Act Carry out the intervention Intervene Plan an action aimed at improving student achievement Page 3 Interpret Insights that answer your question © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector. Analyse data and evidence

Changing and Evaluating Trigger Explore Question Assemble Analyse Interpret Intervene Evaluate Reflect Page 4 Clues found in data, hunches Is there really an issue? What do you want to know? Get all useful evidence together Process data and other evidence What information do you have? Design and carry out action What was the impact? What will we change? © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

The evidence-driven decision making cycle Trigger Explore Question Assemble Analyse Interpret > Intervene Evaluate Reflect Page 5 Clues found in data, hunches Is there really an issue? What do you want to know? Get all useful evidence together Process data and other evidence What information do you have? Design and carry out action What was the impact? What will we change? © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Professionals making decisions How do we decide what action to take as result of the information we get from the analysis? We use our professional judgment Page 6 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Professional decision making We have evidence-based information that we see as reliable and valid What do we do about it? If the information indicates a need for action, we use our collective experience to make a professional decision Page 7 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Professionals making decisions Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge? The answer was Yes. . . so I need to think about how to improve their writing skills. How will I do that? Page 8 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Professionals making decisions Do any particular groups of year 11 students attend less regularly than average for the whole cohort? The analysis identified two groups – so I need to think about how to deal with irregular attendance for each group. How will I do that? Page 9 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Professionals making decisions You asked what factors are related to poor student performance in formal writing. The analysis suggested that poor homework habits have a significant impact on student writing. You make some professional judgements and decide • Students who do little homework don’t write enough • You could take action to improve homework habits – but you’ve tried that before and the success rate is low • You have more control over other factors – like how much time you give students to write in class So you conclude – the real need is to get students to write more often Page 10 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Deciding on an action Information will often suggest a number of options for action. How do we decide which action to choose? We need to consider • what control we have over the action • the likely impact of the action • the resources needed Page 11 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Planning for action • • • Page 12 Is this a major change to policy or processes? What other changes are being proposed How soon can you make this change? How will you achieve wide buy-in? What time and resources will you need? Who will co-ordinate and monitor implementation? © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Planning for action • Is this an incremental change? Or are you just tweaking how you do things? • How will you fit the change into your regular work? • When can you start the intervention? • Will you need extra resources? • How will this change affect other things you do? • How will you monitor implementation? Page 13 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Timing is all • How long should we run the intervention before we evaluate it? • When is the best time of the year to start (and finish) in terms of measuring changes in student achievement? • How much preparation time will we need to get maximum benefit? Page 14 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Planning for evaluation We are carrying out this action to see what impact it has on student achievement We need to decide exactly how we’ll know how successful the intervention has been To do this we will need good baseline data Page 15 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Planning for evaluation • What evidence do we need to collect before we start? • Do we need to collect evidence along the way, or just at the end? • How can we be sure that any assessment at the end of the process will be comparable with assessment at the outset? • How will we monitor any unintended effects? Don’t forget evidence such as timetables, student opinions, teacher observations … Page 16 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

The evidence-driven decision making cycle Trigger Explore Question Assemble Analyse Interpret Intervene > Evaluate Reflect Page 17 Clues found in data, hunches Is there really an issue? What do you want to know? Get all useful evidence together Process data and other evidence What information do you have? Design and carry out action What was the impact? What will we change? © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evaluate the impact of our action Did the intervention improve the situation that triggered the process? If the aim was to improve student achievement, did that happen? Page 18 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evaluate the impact of our action Was any change in student achievement significant? What else happened that we didn’t expect? How do our results compare with other similar studies we can find? Does the result give us the confidence to make the change permanent? Page 19 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evaluate the impact of our action A school created a new year 13 art programme. In the past students had been offered standard Design and Painting programmes, internally and externally assessed against the full range of achievement standards. Some students had to produce two folios for assessment and were unsure of where to take their art after leaving school. The new programme blended drawing, design and painting concepts and focused on electronic media. Assessment was against internally assessed standards only. Page 20 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evaluate the impact of our action • Did students complete more assessments? • Were students gain more national assessment credits? • How did student perceptions of workload and satisfaction compare with teacher perceptions from the previous year? • Did students leave school with clearer intentions about where to go next with their art than the previous cohort? • How did teachers and parents feel about the change? Page 21 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Evaluate the intervention • How well did we design and carry out the intervention? Would we do anything differently if we did it again? • Were our results affected by anything that happened during the intervention period - within or beyond our control? • Did we ask the right question in the first place? How useful was our question? • How adequate were our evaluation data? Page 22 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Think about the process • Did we ask the right question in the first place? How useful was our question? • Did we select the right data? Could we have used other evidence? • Did the intervention work well? Could we have done anything differently? • Did we interpret the data-based information correctly? • How adequate were our evaluation data? • Did the outcome justify the effort we put into it? Page 23 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

The evidence-driven decision making cycle Trigger Explore Question Assemble Analyse Interpret Intervene Evaluate > Reflect Page 24 Clues found in data, hunches Is there really an issue? What do you want to know? Get all useful evidence together Process data and other evidence What information do you have? Design and carry out action What was the impact? What will we change? © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Future practice • What aspects of the intervention will we embed in future practice? • What aspects of the intervention will have the greatest impact? • What aspects of the intervention can we maintain over time? • What changes can we build into the way we do things in our school? • Would there be any side-effects? Page 25 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.

Future directions • What professional learning is needed? Who would most benefit from it? • Do we have the expertise we need in-house or do we need external help? • What other resources do we need? • What disadvantages could there be? • When will we evaluate this change again? Page 26 © New Zealand Ministry of Education – copying restricted to use by New Zealand education sector.