The democratisation of evaluation David Gough Democratisation of

The democratisation of evaluation David Gough Democratisation of Research Methods, ESRC Methods Festival, St Catherine’s College, Oxford, 10 th July 2014 The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London (1) EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC 1 H 0 NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe. ac. uk Web eppi. ioe. ac. uk/

What is democracy? A system of government by the whole population or all the eligible members of a state, typically through elected representatives Oxford Dictionaries A form of government in which all eligible citizens participate equally—either directly or indirectly through elected representatives—in the proposal, development, and creation of laws. It encompasses social, religious, cultural, ethnic and racial equality, justice, and liberty. The term originates from the Greek δημοκρατία (dēmokratía) “ rule of the people“ Wikipedia (2)

Democracy and evaluation 1. The evaluation of democracy – not addressed by this presentation 2. Being democratic about how you evaluate: i. To be more productive (instrumental) ii. To enable those affected to be involved 3. Evaluation as a democratic activity / shared examination of values and data 4. Evaluation activity as a change agent (3)

Issues • Perspectives: • • Ideology, social values, theory and priorities Research questions and methods • Participation • • Engagement Power and control • Research questions and methods required or preferred by particular approaches to democratic evaluation (and the extent of implicit/explicit value specification ) (4)

Four types of DE (J. C Greene 2006) Social values? Research methods? 1. Democratic evaluation: engaging with participants (rather than Bureaucratic by ministries or Autocratic by universities) 2. Deliberative DE: explicitly democratic / ensure that engage with all including weaker groups 3. Participative E: partners in the construction of knowledge / equity 4. Critical E: critical political analysis of macro structural issues (5)

Example 1: Systematic review • PHSE students • Their review – we were technicians • We guessed that their choice would be ‘How to prevent and stop bulling’ • It was in fact: …. . Participative E: Technical methods + user perspectives + user control (6)

Example 2: NICE Guidance • • • Topic for guidance from government Consultation and stakeholder meeting Stakeholder Guidance Advisory Group Commission evidence and develop guidance Explicit social values (SV) policy (e. g. cost effectiveness/equity/rigour/transparency) DE + PE ++: Deliberative process + technical methods + user power + built in SV perspectives + SV of process (7)

Example 3: Participative process evaluation A. Cornwall (2014): IDS Working Paper 2014: 437 To understand the dynamics of change in a nutrition education programme in Kenya 1. Stakeholder analysis: roles & relationships 2. Expectations and experiences of the prog. 3. Analysis by probing +ve and –ve experiences with stakeholders PE of process + exploratory + critique of ‘technocratic evaluation methods (8)

Components of evaluation Not just questions and methods. Also: 1. Perspectives 2. Participation 3. Power and control 4. Other implicit or explicit values Democracy in research involves all of these. Democratic values suggest that they should therefore all be explicit transparent methods and part of evaluation standards (9)

Thank you for your attention Websites EPPI-Centre Website http: //eppi. ioe. ac. uk EIPPEE Website: http: //www. eippee. eu Evidence and Policy: http: //www. policypress. co. uk/journals_eap. asp Twitter @EPPICentre @Prof. David. Gough @EIPPEEnet Email d. gough@ioe. ac. uk The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London (10) EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC 1 H 0 NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe. ac. uk Web eppi. ioe. ac. uk/

An introduction to systematic reviews: Sage Publications Ltd Gough D, Oliver S, Thomas J (2013) Learning from Research: Systematic Reviews for Informing Policy Decisions: A Quick Guide. London: Alliance for Useful Evidence. , Nesta. http: //www. alliance 4 usefuleviden ce. org/assets/Alliance-FUE-reviews -booklet-3. pdf Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews Journal. http: //www. systematicreviewsjournal. com Gough D, (2013) Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Medicine, 11: 22 http: //www. biomedcentral. com/1741 -7015/11/22 (11)