Second Annual Conference on Randomised Controlled Trials in

  • Slides: 16
Download presentation
Second Annual Conference on Randomised Controlled Trials in The Social Sciences: The way forward

Second Annual Conference on Randomised Controlled Trials in The Social Sciences: The way forward The University of York, 13 th - 14 th September 2007 Hosted by ESRC Researcher Development Initiative and the York Trials Unit Speakers represent all areas of social science and public policy and include: Ann Oakley Tom Schuller Bob Slavin Kevin Marsh Paul Dolan Sheila Bird Peter John Further details at http: //www. trialspp. co. uk/ or via email educ-trials-pp@york. ac. uk

What counts as evidence? Stephen Gorard University of York sg 25@york. ac. uk From

What counts as evidence? Stephen Gorard University of York sg 25@york. ac. uk From 1 st August University of Birmingham sgorard@bham. ac. uk http: //www. york. ac. uk/depts/educ/Staff/SGorard. htm http: //trials-pp. co. uk/

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. Cook, T. and Gorard, S. (2007) What counts and what should count as evidence, pp. 33 -49 in OECD (Eds. ) Evidence in education: Linking research and policy, Paris: OECD Gorard, S. (2002) Ethics and equity: pursuing the perspective of non-participants, Social Research Update, 39

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. Gorard, S. , with Taylor, C. (2004) Combining methods in educational and social research, London: Open University Press Sloane, F. and Gorard, S. (2003) Exploring methodological aspects of design experiments, Educational Researcher, 32, 1

Outline of the full cycle of policy research

Outline of the full cycle of policy research

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. Gorard, S. (2003) Quantitative methods in social science: the role of numbers made easy, London: Continuum Gorard, S. (2002) The role of causal models in education as a social science, Evaluation and Research in Education, 16, 1, 51 -65.

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. 4. Part of the problem of research quality lies in traditional research methods training and ‘experts’. Gorard, S. (2005) Revisiting a 90 -year-old debate: the advantages of the mean deviation, The British Journal of Educational Studies, 53, 4 Gorard, S. (2003) Understanding probabilities and re-considering traditional research methods training, Sociological Research Online, 8, 1

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. 4. Part of the problem of research quality lies in traditional research methods training and ‘experts’. 5. Part of the problem of research quality lies in a lack of appropriate use of numbers. Gorard, S. (2006) Using everyday numbers effectively in research: Not a book about statistics, London: Continuum Gorard, S. (2006) Towards a judgement-based statistical analysis, British Journal of Sociology of Education, 27, 1

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. 4. Part of the problem of research quality lies in traditional research methods training and ‘experts’. 5. Part of the problem of research quality lies in a lack of appropriate use of numbers. 6. Part of the problem of research quality lies in an unwillingness to test our cherished theories. Gorard, S. (2004) Three abuses of ‘theory’: an engagement with Nash, Journal of Educational Enquiry, 5, 2 Gorard, S. (2004) Scepticism or clericalism? Theory as a barrier to combining methods, Journal of Educational Enquiry, 5, 1

1. A key ethical concern for those conducting or using publicly-funded education research ought

1. A key ethical concern for those conducting or using publicly-funded education research ought to be the quality of the research, and so the security of the conclusions drawn. 2. It is helpful to consider the research enterprise as a cycle of complementary phases and activities, because this illustrates how all methods can have an appropriate place in the full cycle of research. 3. Working towards an experimental design can be an important part of any research enterprise, even where an experiment is not envisaged or even possible. 4. Part of the problem of research quality lies in traditional research methods training and ‘experts’. 5. Part of the problem of research quality lies in a lack of appropriate use of numbers. 6. Part of the problem of research quality lies in an unwillingness to test our cherished theories. 7. Much of the solution lies in greater scepticism, because the problem is not really one of methods at all. Cook, T. and Gorard, S. (2007) What counts and what should count as evidence, pp. 33 -49 in OECD (Eds. ) Evidence in education: Linking research and policy, Paris: OECD Gorard, S. (2002) Fostering scepticism: the importance of warranting claims, Evaluation and Research in Education, 16, 3, 136 -149.

Contradictory trends Between 1994 and 2004, reports of studies using a researcher manipulated intervention

Contradictory trends Between 1994 and 2004, reports of studies using a researcher manipulated intervention declined from 45% to 33% in the US. But causal statements in non-intervention studies grew from 34% to 43%. In particular, complex statistical approaches such as HLM (MLM) and structural equation modelling were routinely misunderstood by researchers as testing causation. Robinson, D. , Levin, J. , Thomas, G. , Pituch, K. and Vaughn, S. (2007) The incidence of ‘causal’ statements in teaching-and-learning research journals, American Educational Research Journal, 44, 2, 400 -413

Effect of widening participation? Sons born Sons earning Measure of Partial measure parental correlation

Effect of widening participation? Sons born Sons earning Measure of Partial measure parental correlation status Britain 1970 Age 30 Parents 1980+1986 . 27 Norway 1958 Average 1992+1999 Father 1974 . 14 Denmark 1958 -1960 Average 1998+2000 Father 1980 . 14 Sweden 1962 Average 1996+1999 Father 1975 . 14 Finland 1958 -1960 Average 1995+2000 Father 1975 . 15 Country Blanden, J. , Gregg, P. and Machin, S. (2005) Intergenerational mobility in Europe and North America: Report for the Sutton Trust, London: Centre for Economic Performance

Result of academisation? Df. ES - We expect that all Academies will make steady

Result of academisation? Df. ES - We expect that all Academies will make steady upward progress… Good teaching, excellent facilities and motivated pupils will deliver real improvements in educational standards. BBC - The government has released GCSE figures from three of its new flagship Academies in England. All the schools, which were set up in deprived areas, showed remarkable improvements in results. Bexley Business Academy 1997 1998 1999 2000 2001 2002 2003 Percentage 53 eligible for FSM 49 52 50 49 46 42 GCSE points per candidate 23 20 20 22 23 -

An effect of education? p. 238 – ‘The data are based on the entire

An effect of education? p. 238 – ‘The data are based on the entire population of schools… Consequently, the results can be considered as representative of the whole population of pupils who take both KS 3 and GCSE exams’. Multi-level model of the effect of single-sex teaching. p. 246 – ‘The effect of school type is highly significant’ Malacova, E. (2007) Effects of single-sex education on progress in GCSE, Oxford Review of Education, 33, 2, 233 -259

Impact of feedback? Using national data and multi-level modelling to compare the progress of

Impact of feedback? Using national data and multi-level modelling to compare the progress of schools given performance feedback by NFER. p. 102 – ‘Although no actual samples have been drawn… Statistical checks were carried out and no significant difference between the groups was found’ [i. e. in existing background measures]. p. 103 – ‘Schools who participated in PASS showed a significant difference (p<0. 05) in attainment compared to those who received feedback as part of another project. Hammond, P. and Yeshanew, T. (2007) The impact of feedback on school performance, Educational Studies, 33, 2, 99 -113

A school effect? Looks at successive years of positive value-added scores in one England

A school effect? Looks at successive years of positive value-added scores in one England district, 1993 -2002. An improving school is one where VA increases in successive years p. 271 - ‘the pupil intake and time trend explanatory variables included in the fixed part of the value-added model (Model A) were statistically significant (at 0. 05 level)’. p. 261 – ‘it appears that only one in 16 schools managed to improve continuously for more than four years at some point over the decade in terms of value-added’. Thomas, S. , Peng, WJ. And Gray, J. (2007) Modelling patterns of improvement over time: value-added trends in English secondary school performance across ten cohorts, Oxford Review of Education, 33, 3, 261 -295