Design research in statistics education Arthur Bakker Freudentha

Design research in statistics education Arthur Bakker Freudentha. I Institute Utrecht University 2017 summer school

Science fiction faction • Most research is about education as it is • Some about as it was Design research is about education as it could be Bakker, A. (forthcoming). Design research in education: A practial guide for early career researchers. Abingdon, UK: Routledge

Different names • • • Developmental research Design-based research Design experiments Design studies Design or development of instructional materials, environment, is essential part of the research

Why design research? Different types of questions different research methods Is teaching programme A better than B? comparative method What are characteristics of X? Why …? case study, ethnography, survey study… How can we change Y? How can students learn Z? something needs to be designed, tried out and mostly revised

New course for learning, say, statistics If you want to know if it is better than another method, or prove it is better than (quasi-) experimental method with hopefully significant differences But if you want to develop/design a new course and understand how instructional activities work and why, then design research

Example of design research Thesis Design research in statistics education (2004; age group: 11 -13) Vmbo: preparing to vocational education (60%) Havo: pre-higher vocational education (20 -25%) Vwo: pre-university track (15 -20%) Five teaching experiments in grade 7 (11 -12) One in grade 8 (12 -13)

Starting point Changes in society • More intensive use of Information Technology (huge amounts of data) • Need for statistical literacy Not only for employees, but also for critical citizens

Statistics education Disappointing in most countries - Set of unrelated techniques - Students can calculate mean but do not use it for reasoning about differences in groups - Little use of statistics software

Aim and question Aim: contribute to an empirically grounded instruction theory for early statistics education One question: How can students learn about distribution in relation to sampling and other statistical key concepts (data, center, variation, model, . . . )

How to design activities? How do we get or design useful instructional activities? How do you order them? How can RME theory help us?

Phenomenon and concepts Phenomena beg to be organised by thought objects: concepts, techniques, graphs For example: we want to know how much fruit there is on a tree Method: count fruit on average branche and calculate by estimate of number of branches

Phenomenology • How do phenomena lead to concept use? And how are concepts used to understand new phenomena? We can study this from a - Mathematical, - Historical, or - Didactical perspective

Why an historical study? what you can learn from an historical study for the design of instructional materials? Bakker, A. , & Gravemeijer, K. P. (2006). An historical phenomenology of mean and median. Educational Studies in Mathematics, 62(2), 149 -168.

Historical phenomenology What phenomena in the past have led to the development of which statistical concepts, graphs and techniques? Oldest: estimation of large numbers - Men in a fleet (war context) - Number of years between first and last Egyptian king (historian Thucydides) - Height of city walls (to protect the city) to climb it with ladders (again a war context)

How tall should the ladders be? 450 BC

Historical solution for this phenomenon • Many soldiers counted the number of stones vertically • They estimated the height of a stone • The most common number was used (mode): most reliable

Why an historical study? What can you learn from such examples? What are implications for a didactical phenomenology? Should you always study the history?

Different context How many elephants in this herd in Kenia? Estimation? Student strategies?

Strategies of 11 -year-olds: - Add unequal groups (10+15+…) - count equal groups (15 x 20 = 300) - Mr Bean method: height times width: 30 x 18 = 540 - ‘average’ box times #boxes: 42 x 8 = 336

Average box is not too full, not too empty. Roughly in the middle. Implications of too high or low estimation.

Hypotheses (in HLT) • This invites students to think qualitatively about average • In other classes the same strategies • Possibilities to refine notion of average: what is an average box? • Compensation, balance, levelling out, somewhere in the middle…

Results In all classes we observed similar student ideas and strategies Which were useful in subsequent lessons, e. g. estimating averages of distributions in graphs or matrices (say value-bar chart)

Not always in one attempt Elephant estimation went well from the first teaching experiment onwards But most activities needed some adjustments, or were thrown away Example on sampling

London: Westminster Abbey

Pyx chamber in Westminster Abbey: each day one coin put into the box

After many months box was opened, coins weighed and some melted

Need for a new activity Hypothesis: Trial of the Pyx could be useful to let students think about sampling Question in class: how do you check the coin makers? Spies, way the coins, make a smart computerised machine in this historical context discussion was not a success: history an extra obstacle Idea thrown away => try something else

Macro-cycles • • Preparatory phase, design of actvities Teaching experiment Analysis Revision and preparing next cycle Teaching experiment Analysis etc.

Quote Freudenthal Developmental research means: experiencing the cyclic process of development and research so consciously, and reporting on it so candidly that it justifies itself, and that this experience can be transmitted to others to become like their own experience.

HLT: hypothetical learning trajectory A hypothetical learning trajectory is made up of three components: the learning goal that defines the direction, the learning activities, and the hypothetical learning process – a prediction of how the students’ thinking and understanding will evolve in the context of the learning activities (Simon, 1995, JRME)

HLT: useful methodological instrument? Explain why an HLT is a useful methodological instrument in all phases of design research

Use of HLT 1. Make the design and all expectations explicit, and open to scrutiny so they can be tested (often revisions during this phase already). Also for preparing teachers and research assistants 2. Guideline for teachers and researchers during teaching experiments what to focus on (p 40) 3. Compare prediction and actual results HLT more general than concrete activities, and adjustable to new situations

Alternatives to HLTs • Design principles (van den Akker, 1999) • Conjecture mapping (Sandoval, 2014)

Discussion Comments, Questions?

Question How can we ensure that results of design research are • Reliable • Valid • Generalisable?

Reliability versus validity For a test all students get a 7 (out of 10) Very reliable (repeatable), not valid You don’t measure what you want to measure; no justice to research object; no useful knowledge

Reliability Internal: does someone else in the team arrive at the same conclusion? • Data collection (audio, video, student work, observations…) • Systematic coding, peer examination External: replicable? • Can someone follow argumentation? Trackability / transparency of researcher’s learning process

Validity Internal: quality of data and arguments • Ways of data collection • Triangulation • Method of analysis External: generalisability • Are results and theory useful in other contexts? Ecological validity: in real classrooms

Design and experiment Experimental design vs Design experiment What do the words design and experiment mean here? Title of a paper: “Design experiment: Neither design nor experiment”