Using Student Assessment Results to Improve Teacher Knowledge

Using Student Assessment Results to Improve Teacher Knowledge and Practice NESA FLC 24 - 27 October 2013 MARTINA BOVELL, B. A. Dip. Ed. Grad dip. Arts (UWA) Senior Research Fellow

www. acer. edu. au 80 years experience Independent Not for profit Over 300 staff in seven offices Melbourne, Sydney, Brisbane, Adelaide, Perth, India (New Delhi), UAE (Dubai) Four goals: Learners and their needs every learner engaged in challenging learning opportunities appropriate to their readiness and needs The Learning Profession every learning professional highly skilled, knowledgeable and engaged in excellent practice Places of learning every learning community well resourced and passionately committed to improving outcomes for all learners A Learning Society a society in which every learner experiences success and has an opportunity to achieve their potential

ACER and international assessment Member of the International Association for the Evaluation of Educational Achievement (IEA). Consortium leader for the OECD’s Programme for International Student Assessment (PISA) 1998 – 2012.

Purposes of assessment • clarify educational standards • monitor trends over time • evaluate the effectiveness of educational initiatives and programs • ensure that all students achieve essential skills and knowledge

International Schools’ Assesssment http: //www. acer. edu. au/tests/isa Over 64 000 students from 312 schools participated in the ISA in October 2012 and Feb 2013.

The ISA Grades 3 – 10 • • Mathematics Reading Expository writing Narrative writing Grades 8 – 10 • Science online • New in 2013 -2014 testing cycle

How ISA results are reported • All students who participate in ISA tests have their performance measured against a single scale. • There is a separate scale for each of the five domains being assessed (Reading, Mathematics, Narrative Writing, Expository Writing, and Science). • The ISA scale score for a student is different from the “raw” score that the student would get by adding up the number of correctly answered questions on a test. • The ISA scale allows meaningful comparisons of results between different grade levels and between different calendar years even though the tests administered are not the same. • The ISA scales are based on those developed for the Organisation for Economic Co-operation and Development’s (OECD’s) Program for International Student Assessment (PISA).

Types of information Whole grade mean and distribution – all students, language background, gender – this year and previous years (longitudinal comparisons) • compared to all other ISA schools • compared to all other Like schools (based on %s of students with ESB) • For years 8 – 10, comparison with PISA results (Maths, Reading, Science) Classes within whole grades – class mean and individual student scale scores – performance by item classification within a domain • compared to all other ISA schools • compared to all other Like schools • compared to other classes in your school Item by item results for each student – By item classification – compared to other individuals – compared to all other ISA schools

ISA Reports • Paper-based • Electronic • tracking • interactive Science: online test delivery means faster turn-around of results

“Improve teacher knowledge and practice” A growth model: – acknowledges that each student is at some point in their learning – expects every student to make excellent learning progress regardless of their starting point – assesses growth over time

ACER National School Improvement tool (NSIT) • Means of ACER’s mission of improving learning • Endorsed by Australian federal and state governments • Research-based • Informs and assists schools’ improvement agendas • Available to all schools

http: //www. acerinstitute. edu. au/home Part of ACER’s Institute of Learning larger school improvement project. • The Institute’s services include – Professional learning – School review services – Capacity building for school improvement

www. acer. edu. au/documents/NSIT. pdf

9. Schoolcommunity partnerships 8. Effective pedagogical practices 7. Differentiated teaching and learning 1. An explicit improvement agenda 2. Analysis and discussion of data National school improvement tool 6. Systematic curriculum delivery 5. An expert teaching team 3. A culture that promotes learning 4. Targeted use of school resources

Review of a domain

Domain performance levels

Domain performance level: LOW • Teachers do not systematically analyse test and other data for their classes and teachers make little use of data to reflect on their teaching.

Domain performance level: MEDIUM • An ad hoc approach exists to building staff skills in the analysis, interpretation and use of classroom data

Domain performance level: HIGH • One or more members of staff have been assigned responsibility for implementing the annual plan, analysing the full range of school data, and summarising, displaying and communicating student outcome data for the school. The school has ensured that appropriate software is available and that at least these assigned staff have been trained to undertake data analyses. • Time is set aside (e. g. on pupil free days and in staff meetings) for the discussion of data and the implications of data for school policies and classroom practices. These discussions occur at whole-school and team levels.

Domain performance level: OUTSTANDING • Data are used throughout the school to identify gaps in student learning, to monitor improvement over time and to monitor growth across the years of school. • A high priority has been given to professional development aimed at building teachers’ and leaders’ data literacy skills. • Staff conversations and language reflect a sophisticated understanding of student assessment and data concepts (e. g. value-added; growth; improvement; statistical significance). • Teachers are given class test data electronically and are provided with, and use, software to analyse, display and communicate data on individual and class performances and progress, including pre- and post-test comparisons. • Teachers routinely use objective data on student achievement as evidence of successful teaching.

Data literacy framework US Dept of Education office of planning, evaluation and Policy development (2011). Teachers’ ability to use data to inform instruction: challenges and supports. • STUDY: • investigation of teachers’ thinking about student data by administering hypothetical education scenarios accompanied by data displays and questions to individual and small group interviews to teachers and schools identified as exemplary in active data use (50 teachers, 72 small groups) • PURPOSES: • 1. to investigate teachers thinking and reasoning independently about data and how they build on each other's understandings when working with data in small groups. • 2. to isolate the difficulties, misconceptions and support needed.

What they found • Teachers’ likelihood of using data is affected by how confident they feel about their knowledge and skills • Working in small groups appears to promote teachers’ engagement with data. Compared when working individually, teachers were – more likely to arrive at sound data interpretations – more likely to use a wider range of skills when making decisions about how to use and interpret data – able to clarify and frame problems and correct data interpretation errors – more likely to enjoy discussing data

Data literacy skill areas: US study framework • Data location – find relevant pieces of data in the data system or display • Data comprehension – understand what the data is saying • Data interpretation – figure out what the data mean • Instructional decision making – select an instructional approach to address the situation identified in the data • Question posing – frame instructionally relevant questions that can be addressed by the data

Bias when making decisions using data Representative /Availability bias • When judging the probability of something, we use some preconceptions based on how similar two things are or the extent to which an event matches our previous experience. • e. g. irrelevant personal characteristics or stereotypes Anchoring and adjustment bias • Make a decision based on an initial calculation without following though on the calculations. • We ignore data that doesn’t agree with our preliminary decisions or biases (we agree with what we expect). Confidence in making the decision is not always associated with quality of decision making Group decision making can mitigate some of these biases.

PISA findings about decision making PISA in focus #26 http: //www. oecd. org/pisainfocus/ • “Countries vary in the way they use marks, but they all tend to reward the mastery of skills and attitudes that promote learning. ” • “Teachers tend to give girls and socio-economically advantaged students better school marks, even if they don’t have better performance and attitudes than boys and socio-economically disadvantaged students. ” • “It seems that marks not only measure students’ progress in school, they also indicate the skills, behaviours, habits and attitudes that are valued in school. “

What biases?

Narrative Writing assessment Narrative Task Same tasks and marking criteria for all grades – Content 0 - 14: quality and range of ideas, development of plot, characters and setting, the writer’s sense of audience and purpose, the overall shape of the writing. – Language 0 - 14: sentence and paragraph structure, vocabulary and punctuation, and the writer’s voice. – Spelling 0 - 11: considers phonetic and visual spelling patterns and the kind of words attempted, and correctness.

Reporting Content: 7 (max score 14) Language: 6 (max score 11) Spelling: 5 (max score 11) Individual student report comment: Level 5: Write a story with some developed detail in content and using a variety of sentence forms. Spell correctly many words from a student-level vocabulary.

Reading Framework The ISA definition of Reading literacy derives from PISA: Understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society. Goes beyond decoding and literal comprehension and recognises the full scope of situations in which reading plays a role in the lives of students from grades 3 to 10. Three parts: ASPECT TEXT TYPE TEXT FORMAT

Reading Framework Retrieving Information: Locating , selecting, and collecting one or more pieces of information in a text. Interpreting texts: Making sense of the text, constructing meaning making connections and drawing inferences from one or more parts of a text, e. g. cause and effect, compare and contrast, category and example. Involves information that is not stated. Reflecting: Drawing on knowledge, ideas and values external to the text to evaluate a text; relating a text to one’s experience, knowledge and ideas. Continuous text format composed of sentences that are, in turn, organised into paragraphs. These may fit into even larger structures such as sections, chapters and books. Narrative pieces, exposition, description, argument and instructional passages Non–continuous text format Essentially, texts composed of one or more lists in which information is presented in, e. g. tables, graphs, maps and diagrams

Feedback from EARCOS • This workshop was exactly what was needed to guide us. It really helped the focus and the analysis of data. I hope ISA continue to provide this service to support schools who use this service. • Very useful and I am motivated to do my work better. • I felt that the workshop has enabled me to speak with some authority on how we can unpack the ISA results. More importantly I have an insight into how we can use these results to better inform further decisions, ones that are based on student achievement. • Very useful. More of these sessions need to be provided to help teachers understand how data can be used to improve learning. • Informative and well worth the time. Hands on, interactive learning.

ISA Interactive diagnostic report A guide to using the interactive diagnostic report that was demonstrated during the conference session is downloadable at http: //www. acer. edu. au/documents/ISA_Using_the_Interactive_Diagnostic_ Report. pdf

Interrogating data Focus on the student • Did the student perform as well as expected? • Does the performance match expectations/reflect teacher judgement about the student? • What does the student’s response pattern show about the strengths of the student? • What does the student’s response pattern show about the areas of concern for the student? • Are any areas of concern preventing the student from making progress? What might account for these?

Interrogating data Focus on the group – group scores • How does the group achievement relate to the bands? • How does the class distribution against the bands match expectations about the group? • Did the group as a whole perform as well as expected? • Does the relative order of students match expectations about the students? • What students have achieved higher than expected, or lower than expected, in relation to others? • Are there students in the group with similar achievements?

Interrogating data Focus on the group – group scores • Do students with similar scale scores have similar or different response patterns? • What assessment criteria do students perform well on? • What assessment criteria do students perform less well on? • What does the group’s response pattern show about its strengths? • What does the group’s response pattern show about its areas of concern?

Interrogating data • Focus on the teaching program • Has any teaching impacted on the group’s results? • Are there any areas of concern preventing the whole group from making progress?

www. acerinstitute. edu. au/conferences/eppc Presented by practitioners, for practitioners. ACER recognises that, every day, teachers and school leaders are responsible for improving learning among students. This conference provides an opportunity to report on and celebrate the improvements you have achieved within your classes, across the whole school or within networks of schools. Call for papers now open

School Improvement Tool Contact Robert Marshall Senior Project Director Australian Council for Educational Research 19 Prospect Hill Road, Camberwell, Victoria Australia 3124 Robert. Marshall@acer. edu. au • +61 3 9277 5346 • 0439 665 965

Martina Bovell Senior Research Fellow Australian Council for Educational Research 7/1329 Hay Street, West Perth Western Australia 6005 martina. bovell@acer. edu. au +61 8 9235 4821 +61 439 926 277

References • Barber, M. and Mourshed, M. (2007). How the world’s best-performing school systems come out on top. Mckinsey and Co. • Dweck, C. S. (2006). Mindset: The new psychology of success. New York: Balantine Books. • Fullen, M. , Hill, P. , Crevola, C. (2006). Breakthrough. California: Corwin Press. • International Baccalaureate Organisation. (2010). IB learner profile booklet. www. ibo. org • International Baccalaureate Organisation. (2010). Programme standards and practices. www. ibo. org • Masters, G. (2013). Towards a grown mindset in assessment. ACER Occasional essays – 2013. www. acer. edu. au • Tversky, A. and Kahneman, D. (1982). Judgement under uncertainty: Heuristics and biases. In Judgement under uncertainty: Heuristics and biases, eds. D Kahneman, P. Slovic and A. Tversky, 3 -20. NY: Cambridge University Press – cited in US Dept of Education office of planning, evaluation and Policy development (2011). Teachers’ ability to use data to inform instruction: challenges and supports.