NERA 2019 Education in a globalized world University

NERA 2019 Education in a globalized world University of Uppsala 6. 3 -8. 3. 2019 NERA Network 21 Politics of Education and Education Policy Studies The uncertain directional value of data in education Or – what is the value of data for everyday education, its policy and practice? Or – for what, exactly, is data relevant? Jón Torfi Jónasson School of Education, University of Iceland jtj@hi. is http: //uni. hi. is/jtj/en/

The line of argument (1) A. Data is not only important, it is often crucial for a number of reasons. The importance of data can hardly be overstated when placed in the appropriate context. (Note the glossing over here of important semantic differences, such as between data, measurement, indicator and evidence. ) B. There are often several well-known problems with data, in particular as indicators or measurements may lack the validity we often assume they have. This may dramatically diminish the value of data, but this serious issue is not in focus in the paper. We assume we are talking about “good” data. C. There are three very noticeable, forceful and related trends we see in education related to data. D. I. One is the enormous and growing availability of all sorts of data, numerical, digital, verbal, acoustic, visual, …. There all sorts of data explosions. II. Second is the formal policy directives that emphasise the importance of data, ranging from descriptions of the basic functioning of education systems (ILSAs) and how it responds to its clientele, to the daily operations within the classroom (tests, continuous assessment). III. The third is the corollary of both of these, i. e. , the belief that little (nothing? ) sensible can be done without data. For example we have the emphasis on evidence based policy and assessment for learning and it is emphasised that actions on all levels of the education system shall be based on data. But the question we raise is this. Assuming that we have very solid – good data, how much explicit directional value does this data really have, - on its own, e. g. , for the policy maker or the teacher? Jón Torfi Jónasson NERA Uppsala 2019 -On data 2

The line of argument (2) E. F. The question demands a careful analysis, because it is absolutely crucial in our data clouded or dominated world do understand the value of data. Implicit in the question are a number of sub -questions and the overarching question can be re-phrased in a number of ways. 1. Does the data, as such, tell you that something must be done? (The answer we will give, is No, it does not – but paired with our values it certainly may. Thus the values demand a lot of work and attention – more than should be allotted to the data. ) 2. Does data at hand imply that some particular aims – or sub-aims should be set? (The answer we will give, is No, it does not. So the aims demand attention – which is not dominated by data. ) 3. Once some aims have been set and it has been decided that something should be done, does the data tell which route should be selected? Does the data tell you where to go next? (The answer we will give, is No, it does not. ) 4. In other words, what does the data in fact tell you? And thus what is the explicit justification for terms like “evidence based policy / practice” or “formative assessment” – or “assessment for learning”? (I am sure it is already noted that the issue raised is taken here to apply to all levels of the educational discourse. The concern is of course not new but perhaps its considerable urgency. ) Given the suggested answers, a new question arises, i. e. , what has to be added to the setting in order to obtain an answer favourable to data? That is something that is normally implicit within the educational discourse, often felt to be unproblematic, but can be made explicit as will be suggested, and thereby - should - become immediately problematic. Jón Torfi Jónasson NERA Uppsala 2019 -On data 3

The line of argument (3) G. Our mode of thinking about the value of data seems to be moulded by the analogy with a mechanical feedback system, e. g. a temperature regulator and a host of similar devises. When the temperature goes down the heating system is turned on – a simple feedback loop is being harnessed. Such analogy is seriously misleading. Or if you like – we assume that if we know where we stand, what the situation is like, we know where to go and even more importantly we know how to get there – we know how to react. But none of this is the case, in education, if all we have is data. Summary of the claims G. And thus data does not solve all our problems in our daily world of education. In particular, it very rarely tells you where to go. Its directional value ranges from being limited to being non-existing. H. Even when we know that we have to move and even know where to go, data may not tell which is the best way to get there. But it often appears to do so and the rhetoric seems to rely on this. Main conclusion G. Therefore data must be given its proper place within the educational discourse. Data is important but not in the way that is often stated or implied. This applies to the wide spectrum of the educational policy edifice. It also applies to the testing and measurement operational mode, and that of accountability. It also applies to daily teaching. (A similar discussion is necessary for the value of research for the direction of action. ) Jón Torfi Jónasson NERA Uppsala 2019 -On data 4

Coming back to data – we are becoming totally immersed in data – how do we manage? 1. 2. A multitude of scales, diagnostic and discipline specific tests and assessments for a great variety of reasons 1. Related to competence and standing of students, teachers, schools and whole systems (ILSA’s) 2. For teaching purposes, in particular a variety of formative tests and portfolios 3. Related to the absolute and comparative progress of students for a dialogue with parents A variety of new scales 1. In particular a variety of bio-social measures, even continuous monitoring, ranging from stress hormones, breath-analysis, voice or eye-movement monitoring to brain scans) – 2. A variety of well being scales, OECD (How is life? 2017), PISA 2015 (Volume III, Student well-being) 3. PISA, new scales: problem solving (2012), collaboration (2015), global competence (2018), creative thinking (2021), foreign language skills (2024), use of the internet (2027) 3. Continuous extensive collection of data in Artificially intelligent tutoring systems Lundahl, Lisbeth; Arnesen; Anne-Lise & Jónasson, Jón Torfi. (2018). Justice and marketization of education in three Nordic countries: can existing large-scale datasets support comparisons? Nordic Journal of Studies in Educational Policy Volume 4, 2018 Issue 3: Comparative perspectives on Nordic education policy. Pp 120 -132 Published online: 11 Dec Jón Torfi Jónasson NERA Uppsala 2019 -On data 7 2018 https: //doi. org/10. 1080/20020317. 2018. 1542908

There is clearly an official – local and global - emphasis on measurement and visibility UNESCO The Global Education Monitoring Report. Accountability in education Proposed post-2015 education goals: Emphasizing equity, measurability and finance. World Bank World Development Report 2018: Learning to Realize Education's Promise The policy actions they emphasize to address the Learning Crisis, are e. g. • Assess learning, to make it a serious goal. • Act on evidence, to make schools work for learners. Jón Torfi Jónasson NERA Uppsala 2019 -On data Chapter 4: To take learning seriously, start by measuring it - (p. 91) • The learning crisis is often hidden—but measurement makes it visible • Measures for learning guide action • Measures of learning spur action • Choose learning metrics based on what the country needs • Will learning metrics narrow the vision for education? • Six tips for effective learning measurement • Spotlight 3: The multidimensionality of skills 8

Each of us has or knows endless examples of data that is meant to help us, even lead us. Albæk, Karsten; Asplund, Rita; Barth, Erling; Lindahl, Lena. (2015). Youth unemployment and inactivity: A comparison of school-to-work transitions and labour market outcomes in four Nordic countries. Copenhagen: Nordisk Ministerråd, 2015 urn: nbn: se: norden: org: diva-4071 Another distinct feature is the conspicuously similar share of the NEET population across the four countries despite remarkable cross‐country differences not least in the share of young people still lacking an upper secondary degree when aged 21. (p. 277). 9 Jón Torfi Jónasson NERA Uppsala 2019 -On data

Turn to the classroom – a typical situation for every class teacher in compulsory school in Iceland Jón Torfi Jónasson NERA Uppsala 2019 -On data 10

Um endurgjöf – feedback – ofarlega á baugi um þó nokkra hríð Feedback. There has been extensive research done on studying how students are affected by feedback. Kluger and De. Nisi (1996)[26] reviewed over three thousand reports on feedback in schools, universities, and the workplace. Of these, only 131 of them were found to be scientifically rigorous and of those, 50 of the studies show that feedback actually has negative effects on its recipients. This is due to the fact that feedback is often "ego-involving", [17] that is the feedback focuses on the individual student rather than the quality of the student's work. Feedback is often given in the form of some numerical or letter grade and that perpetuates students being compared to their peers. The studies previously mentioned showed that the most effective feedback for students is when they are not only told in which areas they need to improve, but also how to go about improving it. https: //en. wikipedia. org/wiki/Formative_assessment#Feedback Hattie og Clarke 2019, p. 1: Teachers’ definitions: The 10 Cs Comments- give comments on the way you are doing something. But students emphasise guidance: Clarification – answering students questions in class Criticism – when you are given constructive criticism “feedback helps me Confirmation – when your are told you are doing it right know where to go Content development – asking about the comment next” (p. 1) Constructive reflection – giving someone positive and constructive reflection on their work Correction – showing what you did right or wrong, which helps you Q: How well does the Cons and pros – someone telling the pros and cons of your work data give that Commentary – they comment on my work guidance Criterion – relative to a standard Jón Torfi Jónasson NERA Uppsala 2019 -On data 14

The current discourse on formative assessment Dylan Wiliam --- Lorrie Shepard https: //gregashman. wordpress. com/2018/08/11/a n-interview-with-dylan-wiliam/ Interview 2018 … teachers need evidence about what their students are thinking in order to make good decisions, and the quality of that evidence is often poor. The really important thing for me is that formative assessment is neutral with respect to curriculum (what we want students to learn) and pedagogy (how we get students to learn). The big idea —what psychologist David Ausubel called the most important idea in educational psychology—is that any teaching should start from what the learner already knows, and that teachers should ascertain this, and teach accordingly. … we now know that when teachers develop their practice of formative assessment, their students learn more, even when learning is measured in terms of scores on externally mandated tests and exams. https: //www. youtube. com/watch? v=fh 4 z. N 9 JLd. VQ Lorrie Shepard (2017). Notes four stages or levels which relate data and teaching: Lorrie A. Shepard (2018). Learning progressions as tools for assessment and learning #1. Data-driven decision making #2. Strategy-focused formative assessment. #3. Socio-cognitive formative assessment. And the one she thinks is the most valuable. #4. Sociocultural formative assessment. But this requires not only rich data but also a “learning progression” which interweaves assessment, developed curriculum and a theory of learning Learning progression. Jón Torfi Jónasson NERA Uppsala 2019 -On data “Learning progressions are one of the strongest instantiations of principles from Knowing What Students Know, requiring that assessments be based on an underlying model of learning. To support student learning, quantitative continua must also be represented substantively, describing in words and with examples what it looks like to improve in an area of learning. For formative purposes, in fact, qualitative insights are more important than scores. By definition, learning progressions require iterative cycles of development so as to build in horizontal coherence among curriculum, instruction, and assessment. ” 16

What might be the problem with formative assessment? 1. Despite the rigour and richness of the socio-cultural approach Lorrie Shepard presents coupled with the notion of learning progression it raises the question of all kinds of additional insights into the student‘s standing that might affect the teacher‘s action. 2. This may then be added to the questions raised about any learning progression suggested, where a number of alternatives might in principle be suggested, The appealing rigour of the idea of a learning progression, may marginalise (silence, hide) a number of innovative alternatives, 1. See e. g. a number of points raised by Wiliam, D. (2018). How Can Assessment Support Learning? A Response to Wilson and Shepard, Penuel, and Pellegrino. Educational Measurement: Issues and Practice, Spring 2018, Vol. 37, No. 1, pp. 42– 44 2. The notion of a specified learning progression may greatly overestimate the necessity of the a mastery of the base or basic concepts of any discipline – this is perhaps one of the serious curricular myths. See e. g. , the ideas proposed by Tom Fox about ed. Ge-ucation. Tom Fox (2013). Evidence for Addressing the Unsolved through ed. Ge-ucating or - Can Informing Science Promote Democratic Knowledge Production? https: //doi. org/10. 28945/1886 „Ed. Ge-ucating is a process aimed to democratize intellectual breakthroughs, replacing more recent assumptions about specialized experts being the only ones who can create new knowledge. …. This article has suggested how a learning society can be defined as a culture in which citizens with little previous training can be supported and guided to work on intellectual unknowns. We can create strategies for engaging citizen’s imaginations that will restructure, replace, or at least alter the templates which educators, researchers, and most problem solvers have been applying since ancient times. ” Jón Torfi Jónasson NERA Uppsala 2019 -On data 17

Thus what should we keep in mind in a data driven culture? 1. Can goals (and sub-goals) be set on the basis of data? As an example, can the aims of education or upbringing be defined by data. No, they cannot. Data, on its own, does not define aims or goals or sub-goals. But they may be specified by criteria that can be translated into data. But when these have been determined, - measurements (or some other data) may define the gap between the present situation and the goal, but nothing else. So the hard question is: What is the role of data in defining goals? 2. Once the gap between the current state and the ideal state has been defined (by specified criteria at each end), is the route thereby explicitly defined? No, definitely not. Only in situations when the teacher has very clear and well defined curriculum and a very clear theory of learning can she feel justified in selecting a particular route. This is rarely the case, but when these factors guide the route to be taken it is due to them and not the data itself. When it becomes clear that data neither is the basis for defining goals nor the route to a goal, a number of serious questions arise: A) What is the assumed relationship between data and action both at the teaching and policy level. B) How should we rephrase conceptions like evidence based policy or formative assessment. C) To what extent does the data, nevertheless, define the discourse and, perhaps inadvertently, the problems and even aims and alternative courses of action? It is clear that available evidence often does not give much help in setting sensible goals or finding the best avenues in a micro or macro educational setting. Sometimes people may think that the problem can be solved by collecting more data. But that help is very uncertain. Jón Torfi Jónasson NERA Uppsala 2019 -On data 18

Each of us has or knows endless examples of data that is meant to help us, even lead us. Albæk, Karsten; Asplund, Rita; Barth, Erling; Lindahl, Lena. (2015). Youth unemployment and inactivity: A comparison of school-to-work transitions and labour market outcomes in four Nordic countries. Copenhagen: Nordisk Ministerråd, 2015 urn: nbn: se: norden: org: diva-4071 Another distinct feature is the conspicuously similar share of the NEET population across the four countries despite remarkable cross‐country differences not least in the share of young people still lacking an upper secondary degree when aged 21. (p. 277). 19 Jón Torfi Jónasson NERA Uppsala 2019 -On data

Turn to the classroom – a typical situation for every class teacher in compulsory school in Iceland Jón Torfi Jónasson NERA Uppsala 2019 -On data 20

Thus what should we keep in mind in a data driven culture? We have suggested that the directional value of data is, to put it mildly, very uncertain. Or – we asked, what is the value of data for everyday education, its policy and practice? It is both implicit and explicit in the argument that the value is uncertain. That is certainly not implying in any way that data is not invaluable for describing the world and understanding it as we do in all our research. But sometimes it is implied that data is a miracle cure for everything – and that is damaging for education. Or – we asked, for what, exactly, in educational practice is data relevant? Thus, under the enormous pressure to collect data and make it central to our educational endeavour, we should be pressed to think very hard about its various uses, and what precedence it should be given over many other fundamentally important tasks in our educational edifice. In particular we should ponder what important tasks are not solved or helped much by data. Jón Torfi Jónasson NERA Uppsala 2019 -On data 21

Thanks Jón Torfi Jónasson NERA Uppsala 2019 -On data 22