Issues and Challenges around Integrating Qualitative and Quantitative

Issues and Challenges around Integrating Qualitative and Quantitative Data Dr Andrew Booth (with Acknowledgements to Professor Angela Harden & Professor James Thomas)

Overview • What is Already Known around Integrating Qualitative and Quantitative Data • Overview of Methodological Issues/Challenges • Recent Developments (with focus on literature of last two years) • Outstanding Issues/Challenges

Drivers • Complexity of Context • Complex Interventions • Complexity of Questions: Feasibility Appropriateness Meaningfulness Effectiveness/Economy (FAME) • Complexity of the Data

Complex Interventions • Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome. ” Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337

FAME! (i. e. Feasibility etc…)

Complex Questions!

When You’re Lost You Need a Map!

Complexity of the Data E. g. Process Evaluations • Elaboration / expansion / explanation • Initiation – Where different data ‘speak’ to one another, we are able to analyse and explain findings / analyse variation • Complementarity – Use all evidence at our disposal • Contextualisation – Some questions are really about re-contextualising findings for specific use • View problem from multiple directions / perspectives

Approaches and methods in 2004… “Quantitising” “Qualitising” • Narrative summary • Thematic analysis • • Grounded theory Meta-ethnography Meta-study Miles and Huberman's data analysis techniques Content analysis Case survey Qualitative comparative analysis Bayesian meta-analysis 2004

Approaches and methods 2015 Bayesian metaanalysis Critical Interpretive synthesis Metanarrative review Narrative synthesis Integration of qualitative and quantitative research Realist synthesis Mixed methods synthesis

Origins of approaches and methods Method Developed by/Exemplars Context and purpose Narrative synthesis Popay et al. (2005) Cochrane remit – to examine issues of process, implementation and experience Mixed methods synthesis Thomas, Harden et al. Informing policy to promote children’s (2004) health Harden and Thomas (2005) Bayesian synthesis Roberts et al. (2002) Factors that affect the uptake of childhood immunisation Critical interpretive Dixon-Woods et al. (2006) synthesis Access to healthcare for vulnerable groups Meta-narrative Greenhalgh et al. (2005) To review research on diffusion of innovation to inform healthcare policy Realist synthesis Pawson (2006) To develop and test theories of change underpinning complex policy interventions

Overview of Methodological Issues/Challenges • Key Issue remains “Integration” • At what point is integration taking place? : – Review Question – Methods – Findings – Synthesis – Discuss – Recommendations

Integration of Methods

Qual/Quan and Mixed

Challenges (Boeije et al, 2014) • Extracting the separate qualitative and quantitative components can be difficult if tightly integrated within different stages of the study, e. g. data collection, data-analysis, and interpretation. • Where components are integrated, components are more concise and narrow in scope. Specifically aim to compare and integrate well -defined qualitative and quantitative components. Turning these studies into MMS as they are meant to be, makes separate components unfit for inclusion in separate research syntheses. • As result of integration, MMS outputs are hard to extract and formulate as newly gained insights. Reflects debate about nature/quality of yield in MMS. • Bottomline: Well integrated primary research makes it more difficult to integrate it within a qualitative evidence synthesis

Different degrees of combining disciplines (Oliver, 2015): • Distinct disciplines with different strengths for different purposes (multidisciplinary); making links between different disciplines and creating additional knowledge from where they meet (interdisciplinary) and dissolving boundaries between them and around them (transdisciplinary). – Multidisciplinary approach: parallel procedures for process evaluations and controlled trials (peer-delivered health promotion), then discussed their findings together, latter help to explain former. Process evaluations revealing how teachers often undermine peer delivery by retaining control explained why controlled trials showed that the interventions were not working, but could not offer solutions. – Interdisciplinary approach: (Barriers and facilitators) - interdisciplinary research, where new learning emerges at the interface. Comparing findings of qualitative studies with others evaluating effects of interventions – some by sound designs and others by flawed designs – revealed appropriate interventions ready for policy consideration or for rigorous evaluation. – Transdisciplinary approach (children and healthy eating) - merging two methodologies to provide single coherent product, single report in widely read medical journal. Findings of overarching synthesis present statistical meta-analysis in terms determined by the synthesis of children’s ‘views’. In final synthesis and conclusion, contributions of experimental designs and qualitative studies are indistinguishable. Methodological development transcended academic disciplines.

What Tools do we have? Methods • Bayesian Synthesis • Critical Interpretive Synthesis • “EPPI-Method” • Meta-Narrative • Narrative Synthesis • Qualitative Comparative Analysis • Realist Synthesis Techniques • Logic Models • Frameworks • Matrices

Review questions • What is known about the barriers to, and facilitators of, healthy eating amongst children? • Do interventions promote healthy eating amongst children? • What are children’s perspectives on healthy eating? • What are the implications of the above for intervention development?

REVIEW PROCESS Searching, screening and mapping Focus narrowed to ‘fruit &veg’ Synthesis 1: Trials (n=33) 1. Quality assessment 2. Data extraction 3. Statistical meta-analysis Synthesis 2: Qualitative studies (n=8) 1. Quality assessment 2. Data extraction 3. Thematic synthesis Synthesis 3: Trials and qualitative studies

Synthesis 3: Across studies Children’s views Recommendation for interventions Trials Good quality Other Do not promote fruit and vegetables in the same way 0 0 Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one 5 5 Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 5 6

Synthesis 3: Across studies Increase (standardised portions per day) in vegetable intake across trials Little or no emphasis on health messages

In focus: Bayesian synthesis Recent examples • Aim is to test theory • Findings from qualitative and Factors affecting the uptake of quantitative research are ‘fused’ childhood immunisation • Only three worked examples to (Roberts et al. , 2002) date Factors that influence • Variation in weight given to the adherence to HIV qualitative evidence medication regimes • Synthesis product is a set of (Vollis et al. , 2009) weighted factors associated (Crandell et al. , 2011) with/predicting the phenomenon under review

Factors that influence adherence to HIV medication (Crandell et al. , 2011)

In focus: Critical Interpretive Synthesis • Aim is to generate theory from large and diverse body of literature • Literature itself is an object of scrutiny (critical) • Comprehensive search to identify sampling frame • Purposive and theoretical sampling • Analysis leads to generation of synthetic constructs and a synthesising argument Examples Access to health care for vulnerable groups (Dixon. Woods et al. 2006) Use of morphine to treat cancer related pain (Flemming, 2009) Nurses response to suicide and suicidal patients (Talseth and Gilje, 2011)

Access to health care for vulnerable groups (Dixon. Woods et al. 2006) • Based on 119 papers • Interpretive qualitative analysis of diverse types of studies • Conceptual and methodological problems with measures of health service utilisation • Synthesising argument organised around a set of central concepts (e. g. navigation, adjudications) with the synthetic construct of ‘candidacy’ at the core

Access to health care for vulnerable groups (Dixon-Woods et al. 2006) Core construct – candidacy Central concepts “Candidacy describes the ways in which people's eligibility for medical attention and intervention is jointly negotiated between individuals and health services……. candidacy is a dynamic and contingent process, constantly being defined and redefined through interactions between individuals and professionals, including how "cases" are constructed. ” • Identification of candidacy • Navigation • Permeability of services • Appearances at health services • Adjudications • Offers and resistance • Operating conditions and local production of candidacy

In focus: Meta-narrative review • Aim is to make sense of and understand diverse bodies of literature and their findings • Literature itself is an object of scrutiny (critical) • Searching is iterative, ‘snowballing’ a key technique • Analysis leads to production of a set of meta-narratives (‘storylines of research’) Recent examples Spread and sustainability of innovations in health service delivery and organisation (Greenhalgh et al. , 2005) Understanding the use of electronic patient records in health care organisations (Greenhalgh et al. , 2009) See also the: Realist and meta-narrative evidence synthesis evolving standards project (RAMASES) (Greenhalgh et al. , 2011; Wong et al. , 2013)

A meta-narrative approach • The influence of Kuhn’s ‘paradigms’ (1962) and the makings of the first meta-narrative • The essential technique is interpretive synthesis exploring distinct research traditions, each with its own meta-narrative • Methods of ‘unpacking’ the meta-narrative: exploratory methods; expert consultations; snowballing; database searching

Stages of a meta-narrative review (from Greenhalgh et al. , 2009)

Meta-narratives identified in the electronic patient records review (from Greenhalgh et al. , 2009)

Comparing approaches and methods Method Idealist – realist continuum Mixed methods Realist synthesis Deconstruct body of literature? No Mixed methods lens Bayesian synthesis Realist No Critical interpretive synthesis Idealist Yes Alternative or single paradigm stance Meta-narrative Idealist Yes Dialectical stance Complementary strengths stance and Dialectical stance Alternative or single paradigm stance Other characteristics Little iteration in methods Synthetic product aims to directly address policy Iterative approach key Synthetic product requires interpretation

Mixed methods systematic reviews Working definition Combining the findings of ‘qualitative’ and ‘quantitative’ studies within a single systematic review, in order to address the same, overlapping or complementary review questions (Harden and Thomas, 2010)

Three ways in which reviews are mixed…. 1. The types of studies included and hence the type of findings to be synthesised (i. e. ‘qualitative/ textual and quantitative/numerical) 2. The types of synthesis method used (e. g. statistical meta-analysis and qualitative synthesis) 3. The mode of analysis: theory testing AND theory building

Mixed Methods Reviews • Mixed methods reviews have a distinctive heritage – Address complex (and compound) questions – Use different types of evidence in a ‘dialectical’ fashion to grapple with complexity – Mitigate impact of the lack of intervention / evaluation replication – Blend the macro and micro perspective • Some thinking & methodological development has already taken place – in the primary research mixed methods literature

Three main types of integration – Sequential explanatory design – Sequential exploratory design – Convergent design (Pluye and Hong, 2014 – See Next Slide)

Mixed Methods Syntheses (Pluye & Hong, 2014)

Mixed methods synthesis type 1: Sequential explanatory design Synthesis design: a) QUAN synthesis is followed by, and informs, QUAL synthesis; and b) QUAL synthesis helps to explain some results of QUAN synthesis • Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 10101012. (http: //www. bmj. com/cgi/content/full/328/7446/1010)

Mixed methods synthesis type 2: Sequential exploratory design Synthesis design: • a) QUAL synthesis is followed by, and informs, the QUAN synthesis; and • b) QUAN synthesis generalizes or tests findings of the QUAL synthesis • Aim: – Identification of new hypotheses and knowledge gaps (e. g. development of a typology) • Sutcliffe K, Stokes G, O’Mara-Eves A, Caird J, Hinds K, Bangpan M, Kavanagh J, Dickson K, Stansfield C, Hargreaves K, Thomas J (2014) Paediatric medication error: A systematic review of the extent and nature of the problem in the UK and international interventions to address it. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. ISBN: 978 -1 -907345 -73 -9

Mixed methods synthesis type 3: Convergent approaches • Two types: – QUAL data – QUAN data • Data from primary studies transformed before synthesis begins • E. g. Qualitative Comparative Analysis

Logic Model for Integrating Evidence Baxter et al, 2014)

Recent development - Matrix • Candy et al. (2013) Using qualitative evidence on patients views to help understand variation in effectiveness of complex interventions • Used qualitative comparative analysis to identify pathways to effectiveness • Worked example - improving adherence to drug therapy

Recent developments

Future challenges • Need more worked examples • Focus on methods and tools for the actual integration – Enhancing transparency – Establishing rigour • Further conceptual work to illuminate points of difference, strengths and weaknesses, fit for purpose (e. g. aggregation and configuration – Sandelowski et al. , 2011; Gough et al. , 2012) • Learning from, and contributing to, the mixed methods literature for primary research. • Critical appraisal and reporting standards

The Key is Difference = Integration • “for mixed research synthesis to advance, researchers must solve the problems generated by the methodological diversity within and between qualitative and quantitative studies. Difference has recurrently been identified as the most important factor complicating both the qualitative and quantitative research synthesis enterprises”. (Sandelowski et al, 2007)

In Summary • Mixed methods research syntheses have dynamic and evolving ‘heritage’ • Address complex (and compound) questions, interventions, wicked problems • Need to move from “handmaiden” role to a genuine “enhancement” role (Popay; 1998; Petticrew, 2015) • Methods evolving: fast moving, interesting and rewarding field!

References • • • Baxter, S. K. , Blank, L. , Woods, H. B. , Payne, N. , Rimmer, M. , & Goyder, E. (2014). Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions. BMC medical research methodology, 14(1), 62. Boeije, H. , Slagt, M. , & van Wesel, F. (2013). The Contribution of Mixed Methods Research to the Field of Childhood Trauma A Narrative Review Focused on Data Integration. Journal of Mixed Methods Research, 7(4), 347 -369. Candy B, King M, Jones L, Oliver S. Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Med Res Methodol. 2011 Aug 26; 11: 124. doi: 10. 1186/1471 -2288 -11 -124. Hannes K. Building a case for mixed-methods reviews. In: Richards & Hallberg. Complex Interventions in Health. Routledge, 2015. Heyvaert, M. , Maes, B. , & Onghena, P. (2013). Mixed methods research synthesis: definition, framework, and potential. Quality & Quantity, 47(2), 659 -676. Moore, G. F. , Audrey, S. , Barker, M. , Bond, L. , Bonell, C. , Hardeman, W. , . . . & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. bmj, 350, h 1258.

References • Pluye, P. , & Hong, Q. N. (2014). Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Public Health, 35(1), 29. • Pluye P, Gagnon MP, Griffiths F, et al. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud 2009; 46: 529– 46. doi: 10. 1016/j. ijnurstu. 2009. 01. 009 • Sandelowski et al. (2007) Comparability work and the management of difference in research synthesis studies. Social Science and Medicine, 64: 236 -247 29. • Shaw RL, Larkin M, Flowers P. Expanding the evidence within evidencebased healthcare: thinking about the context, acceptability and feasibility of interventions. Evid Based Med 2014; 19: 201– 3. doi: 10. 1136/eb-2014101791 • Souto RQ, Khanassov V, Hong QN, et al. Systematic mixed studies reviews: updating results on the reliability and efficiency of the mixed methods appraisal tool. Int J Nurs Stud 2015; 52: 500– 1.