Systematic Reviews The Potential of Different Methods Steve

Systematic Reviews: The Potential of Different Methods Steve Higgins Henry Potts James Thomas Geoff Wong ESRC Research Methods Festival Oxford, 5 July 2012 (1)

Introduction • Aims – To outline major approaches to systematic reviewing and research synthesis – To demystify what on first glance looks like a bewildering set of overlapping approaches – To examine some of the assumptions that underpin these approaches – To discuss how these approaches might help in my (your!) situation (2)

Introduction • Session structure: – 20 Minute presentation + 10 minutes discussion / clarification of four approaches to synthesis – Coffee break after the 3 rd / 4 th presentation – Discussion in pairs – Questions + panel discussion (3)

What is a systematic review? • Systematic: ‘done or acting according to a fixed plan or system; methodical’ • Review: ‘a critical appraisal of a book, play, or other work’ (OED) (4)

What is systematic research synthesis? • Synthesis: ‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a theory or system’ (OED) • Not just a report of the findings of the individual studies in a review • Involves a transformation of the data from primary studies in some way • Synthesis of the findings of all the included studies in order to answer the review question (5)

Aims of synthesis reviews • Answer questions: what do we want to know and how can we know it? • Bring together and ‘pool’ the findings of primary research (i. e. clarify what we know) • Any question so potentially any type of evidence • Driven by review users to answer relevant questions in relevant ways (6)

Introduction • We will look at four main approaches to reviewing: – Meta-analysis – The synthesis of ‘qualitative’ research – Realist review – Meta narrative review • Each have different disciplinary backgrounds and underlying assumptions • They all bring together research using two main logics (7)

Aggregation in reviews Aggregation refers to ‘adding up’ (aggregating) findings from primary studies to answer a review question… … to indicate the direction or size of effect … and our degree of confidence in that finding (8)

Configuration in reviews Configuration involves the arrangement (configuration) of the findings of primary studies to answer the review question…. … to offer a meaningful picture of what research is telling us. . . across a potentially wide area of research (9) •

Most reviews contain elements of both, but some patterns are common Aggregation (10) Configuration Approach to theory Test Research question Closed Concepts Secure prior to review Emergent in review Timing of methods decisions A priori Iterative / emergent Search for studies Representative / unbiased / exhaustive Quality appraisal Avoiding bias Explore Generate Open Sufficient / purposive Richness of data

Methods for the synthesis of qualitative research James Thomas 1 ESRC Research Methods Festival Oxford, 5 July 2012 (11) The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London, UK 1

Synthesis of qualitative research: outline • What can qualitative research tell us? • Current developments/approaches in qualitative review methods • Examples of syntheses of qualitative research (12)

What can qualitative research tell us? (13)

Qualitative research… • aims to provide an in-depth understanding of people’s experiences, perspectives and histories in the context of their personal circumstances or settings…* • with the use of unstructured methods which are sensitive to the social context of the study* • the capture of data which are detailed, rich and complex…* • is generalisable on the basis of theory not statistical probability ** * Spencer et al (2003); ** Popay (2006) (14)

Methods used include*… • • • exploratory interviews focus groups observation conversation discourse and narrative analysis documentary and video analysis *Spencer et al (2003) (15)

Is it simply the analysis of textual data? “Increasingly, the terms ‘quantitative research’ and ‘qualitative research’ came to signify much more than ways of gathering data; they came to denote divergent assumptions about the nature and purposes of research in the social sciences” Bryman (1988) Quantity and Quality in Social Research. London: Unwin Hyman. p. 3. (16)

Qualitative research can* • help to define policy questions • look in detail at how a programme or trial was actually implemented • help to determine appropriate outcome measures by looking at ‘subjective’ outcomes • help to clarify what counts as effective or successful (17) * Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p. 36

…continued • identify and explore unintended consequences • contribute to service delivery and policy development by describing processes and contexts • inform and illuminate quantitative studies, e. g. by contributing to the design of structured instruments, assessing the fairness of comparisons in experimental studies, or unpacking variation within aggregated data (18) Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p. 36

Current developments in synthesising qualitative research • Growing recognition of the value of qualitative research to inform policy & practice • Methods developing to facilitate this (19)

(20) Try: http: //www. wordle. net/

Methodological study • We searched for methodological papers concerning the synthesis of concepts or theories – Purposive, rather than systematic search – Reference ‘chasing’ – Google scholar search – Handsearching key journals • 203 papers retrieved • 9 distinct methods for synthesis (21)

Methods for the synthesis of qualitative research identified • • (22) Meta narrative synthesis critical interpretive synthesis meta-study meta-ethnography grounded theory thematic synthesis framework synthesis ecological triangulation

Many methods: similar or different? • Examined the methods across different dimensions: – Epistemology – Approach to quality assessment – Attitudes towards problematizing the literature – Use of review question – How similar / different the included studies were – Characteristics of the synthetic product • Found they fell into two broad camps: ‘idealist’ and ‘realist’ (23)

‘Realist’ approaches Purpose • To answer a policy relevant question Methods • Qualitative/ quantitative data analysed with qualitative/ quantitative methods • Searching linear or iterative • Quality assessment of study methods Product • Directly applicable to policy and practice decisions (24)

‘Idealist’ approaches Purpose • To explore and construct concepts from the data • For generating theory Methods • Qualitative data analysed with qualitative methods • Searching iterative • Qualitative assessment of study content > method Product • Complex, requiring further interpretation before being used for policy or practice (25)

Conclusions • Operationally, many methods are very similar • Underlying principles differ • Product differs in terms of the amount of additional interpretation required. This may reflect… • (Barnett-Page and Thomas (2009)) (26)

Perspectives on how evidence informs the policy process Knowledge-driven model; Problem-solving (policy driven) model* Idealist Realist Less suited More suited Less suited (either researcher disseminates research or policy-maker commissions / requests specific research) Interactive model; Enlightenment model* (ongoing interactions between researcher and policymaker; ‘gradual sedimentation of insights…’) (27) *Weiss (1979)

Two examples • All this is a bit abstract… what do systematic reviews of qualitative research look like? • Two examples: – The ‘fine grain’ of a synthesis – ‘Mixed methods’ synthesis (28)

Example 1: Informing policy about the ‘obesity epidemic’ What are children’s views about: • the meanings of obesity or body size, shape or weight (including their perceptions of their own body size)? • influences on body size? • changes that may help them to achieve or maintain a healthy weight? (29) Views = attitudes, opinions, beliefs, understanding or experiences As distinct from health/weight status, behaviour, factual knowledge Rees et al 2009

‘Thematic synthesis’ • Similar to other methods of synthesising qualitative research (e. g. ‘meta-ethnography’) • • Source data = text (documents) Source material = conceptual Key method = translation Final product = interpretation • (Thomas and Harden 2008) (30)

Stages of thematic synthesis • Stages one and two: coding text and developing descriptive themes – Identifying the ‘findings’ – Line-by-line coding – Developing descriptive themes • Stage three: generating analytical themes – In the light of the review question (31)

Screenshot – line-by-line coding (32)

Screenshot – descriptive codes diagram (33)

Example 2. adding insights to other existing reviews • How might a systematic review of qualitative research support or add insight to a metaanalysis? • Product of qualitative synthesis can form the conceptual framework within which heterogeneity can be explored in a meta-analysis (34)

Review Question e. g. ‘What is known about the barriers to, and facilitators of, outcome X (e. g. physical activity) amongst population A (e. g. young people Searching, screening mapping and user involvement Synthesis 1 Synthesis 2 e. g. Statistical meta-analysis of trials a) Data extraction b) Quality assessment c) Effect sizes pooled d) Narrative synthesis e. g. Thematic synthesis of qualitative studies a) Data extraction b) Quality assessment c) Descriptive themes d) Analytical themes Addresses sub RQ e. g. ‘which interventions are effective? ’ Addresses sub RQ e. g. ‘what are people’s perspectives and experiences ? Synthesis 3 Driven by overall review question Integration of separate syntheses e. g. a) Matches, mis-matches and gaps b) Hypotheses generated in synthesis 2 tested amongst trials in synthesis 1

Cross study synthesis: Findings Children’s views (36) Outcome evaluations Recommendation for interventions Good quality Other Do not promote fruit and vegetables in the same way No soundly evaluated interventions No other interventions identified Brand fruit and vegetables as an ‘exciting’ or childrelevant product, as well as a ‘tasty’ one 5 soundly evaluated interventions identified 5 other interventions Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 5 soundly evaluated interventions identified 6 other interventions identified

Cross study synthesis: an example of sub -group analysis Increase (standardised portions per day) in vegetable intake across trials Little or no emphasis on health messages (37)

This method of synthesis across study types: • preserves the integrity of the findings of the different types of studies • allows us to integrate ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives • allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance – protects against ‘data dredging’ (38)

Assumptions / theoretical models • Mixed methods synthesis can draw on thinking relating to mixed methods in primary research • In terms of a frequently used taxonomy of paradigm stances (Creswell 2011) e. g. – Incommensurability (cannot be mixed) – A-paradigmatic (can be mixed and matched in different ways) – Complementary strengths (not incompatible, but are different and should be kept separate) – Dialectic (paradigms are important in different ways leading to useful tensions & insights) – Alternative paradigm (‘mixed methods’ paradigm; foundation in e. g. pragmatism) (39)

Conclusions • Qualitative research offers valuable insights that can inform policymaking • Methods for synthesising qualitative research are still being developed • Selection of a particular method depends on the type of answer required (and the means by which the review will inform policy) (40)

References • • • (41) Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9: 59. doi: 10. 1186/1471 -2288 -9 -59. (http: //www. biomedcentral. com/14712288/9/59) Creswell J (2011) Mapping the Developing Landscape of Mixed Methods Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods in Social & Behavioral Research. New York: Sage. Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. London: Sage Thomas J, Harden A (2008) Methods for thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8: 45 doi: 10. 1186/1471 -2288 -8 -45 (http: //www. biomedcentral. com/1471 -2288/8/45) Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 10101012. (http: //www. bmj. com/cgi/content/full/328/7446/1010)

Systematic Reviews: The Potential of Different Methods Steve Higgins Henry Potts James Thomas Geoff Wong ESRC Research Methods Festival Oxford, 5 July 2012 (42)

Summary (43) Heterogeneity Aggregation / configuration Meta-analysis A problem in fixed effect Allowance made for in random effects For ‘exploration’ Fixed & random effects models are pure aggregation Sub-group & metaregression both aggregate & configure Synthesis of qualitative research Expected. Key challenge is judging when contexts / concepts are too dissimilar to be compared. Similar concepts are ‘aggregated’; most synthesis involves configuration Realist review Necessary to obtain different Aggregates within CMO contexts etc configurations; configures between them Meta narrative review Necessary to obtain diversity Aggregation within in paradigms; configuration between them

Your turn Research question 1 Research question 2 (44) Heterogen- Metaeity? analysis Qualitative synthesis Realist review Meta narrative review