Extensions to traditional statistical metaanalysis James Thomas Systematic

Extensions to traditional statistical meta-analysis James Thomas Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10 th July 2014 The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London (1) EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC 1 H 0 NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe. ac. uk Web eppi. ioe. ac. uk/

Outline • What is meta-analysis, and why are ‘extensions’ needed? • What is complexity? And why is it a challenge in systematic reviews? • Extensions to traditional meta-analysis when complexity is encountered (2) • Acknowledgement: presentation draws on: O’Mara-Eves A, Thomas J (2013) Methods for configurational synthesis: extensions to traditional meta-analysis for addressing intervention complexity and contextual variation in reviews. 21 st Cochrane Colloquium: Quebec 19 -23 September

The common stages of a systematic review; focus here on synthesis Form review team (involve ‘users’) Formulate review question, conceptual framework and inclusion criteria (develop ‘protocol’) Search for and identify relevant studies Describe studies Assess study quality (and relevance) Synthesise findings Communicate and engage (3) Synthesis Map

Traditional meta-analytic models • Typically are used to address three key research questions: – What is the overall estimate of the size of the effect and its precision? – Is there heterogeneity across the study effects? – What (if any) variables explain differences across the study effects (if heterogeneity is present)? (4)

Aggregative approaches in research Aggregative reviews predominately add up (aggregate) findings of primary studies to answer a review question… … to indicate the direction or size of effect (5)

RCT forest plot: Does children’s participation in structured arts activities improve their cognitive learning outcomes? (6) Newman M, Bird K, Tripney J, Kalra N, Kwan I, Bangpan M, Vigurs C (2010) Understanding the impact of engagement in culture and sport: A systematic review of the learning impacts for young people. London: Department for Culture, Media and Sport. http: //culture. gov. uk/images/research/CASE-systematic-review. July 10. pdf

Are we limiting ourselves? • Current popular meta-analytic methods are limited to: – Questions of differences between two groups or correlations or variables – One causal / relational proposition at a time – One outcome at a time (7)

Whose questions are we addressing? • Policymakers and practitioners usually do not ask a single narrow aggregative question • They begin with a particular problem and ask “what is the best way to achieve outcome X? ” • They also ask “does it vary according to…? ” and “What does X mean to Y? ” • And they ask these questions relating to complex problems (8)

A complex intervention • Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome. ” • Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337 Some would say the above is merely complicated… (9)

Complicated and complex • Truly complex interventions are best conceptualised as dynamic processes – Virtuous circles – Feedback loops – Non-linear step changes in responses / outcomes – Multiple ‘routes’ to effectiveness • Rogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008; 14(1): 29 -48 (10)

Complex reviews of social research: • Start from a given ‘problem’ – often a known outcome and population • Aim to identify a range of possible ‘solutions’ • Rarely aim to come to a single answer • Acknowledge that there are rarely replications of interventions in social research – Use heterogeneity to better understand factors that influence the impact of interventions – Contain detailed and complex conceptual frameworks (programme theories etc) • So to configure findings (as well as aggregate) (11)

Configurative approaches in research • Configurative reviews predominately arrange (configure) the findings of primary studies to answer the review question…. • … to offer a meaningful picture of what research is telling us (12)

Other tools in our toolbox • Sub-group analysis • Meta-regression • Network metaanalysis • Multivariate metaanalysis • Path analysis • Factor analysis • Qualitative comparative analysis • …? (13)

Subgroup analysis (14) Di. Censo et al (2002) BMJ; 324: 1426

Meta-regression (15) Catalá-López et al. BMC Psychiatry 2012, 12: 168

Network meta-analysis • Facilitates an estimate of the relative effectiveness of interventions – even when they have not been directly compared with one another in a trial (16) Thorlund and Mills Systematic Reviews 2012, 1: 41

Problems with ‘traditional’ approaches • The above approaches work well in some situations, BUT • There are rarely replications of complex interventions (is it possible to have a genuine replication? ); leads to a lack of data • Even when an analysis has many studies, interventions, contexts etc. all differ – Lots of unexplained heterogeneity • Symmetrical nature of correlational analysis (17)

Complexity and correlational analysis • Correlation is symmetric – When testing for a connection between cause and effect, also tests equally for absence of cause and absence of effect • Correlation therefore cannot detect multiple causal pathways* – E. g. Asserting that ‘interventions which are delivered by peers tend to be effective’ should not require that those not delivered by peers are not effective * There are usually too few studies to use interaction variables (18)

Another approach: qualitative comparative analysis (QCA) • Originally developed by Charles Ragin in political science and historical sociology • Not a correlational approach • Useful for small numbers of studies • Focus can be on combinations of intervention components • More inductive mode of analysis than above (19) Suggests that intensity & quality are sufficient to gain effective outcome Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67

Summary • Meta-analysis can address a single focused question, requiring a straightforwardly aggregative answer • “Real world” questions tend to require configuration AND aggregation • Extensions to meta-analysis are able to configure and aggregate study findings successfully • But few current methods can cope with genuinely complex situations (20)

Thank you for your attention Websites EPPI-Centre Website http: //eppi. ioe. ac. uk Twitter @James_M_Thomas @EPPICentre Email j. Thomas@ioe. ac. uk EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC 1 H 0 NR The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London (21) Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe. ac. uk Web eppi. ioe. ac. uk/