An introduction to systematic reviews and metaanalyses Colin

An introduction to systematic reviews and meta-analyses Colin Josephson Assistant Professor of Neurology University of Calgary

Faculty/Presenter Disclosure • Faculty: Colin Josephson • Relationships with commercial interests: – – Grants/Research Support: Nil Speakers Bureau/Honoraria: Nil Consulting fees: Nil Other: Nil

Disclosure of Commercial Support • This program has received financial support from: Nil • This program has received in-kind support from: Nil • Potential conflict(s) of interest: – Nil

“That’s so meta…” • Greek: prefix for ‘after’ or ‘beyond’ • English: abstraction form a concept that is used to complete or add to said concept • Unfortunately now a ‘hipsterism’

Pyramid of evidence

A lot of work…but for what? • Traditional ‘narrative’ reviews are at high risk of bias • Systematic reviews address a specific question in a standardised, reproducible manner • If low heterogeneity (statistical inconsistency) then aggregating data is possible

What we want to avoid… Publication bias (selective reporting) Cochrane Handbook

The ideal result Cochrane Handbook

Normal distribution

Sampling error

The ‘systematic’ approach PICOST JAMA Users Guide to the Medical Literature

Judge a man by his questions… • What was the question of the review? • Was it sensible, clinically relevant, and focused? – PICOST • A narrow, precise question diminishes the risk of heterogeneity, thus providing accurate conclusions

Examples • What is the effect of surgery on epilepsy outcomes (e. g. seizure-freedom and quality of life)? • What is the effect of resective surgery on seizure freedom? • In patients with TLE secondary to MTS (P), what is the effect of an ATL (I) compared to medical management (C) on seizure-freedom (O) in randomised controlled trials (S) measured at one-year (T)

The ‘systematic’ approach Statistical plan (PROSPERO) JAMA Users Guide to the Medical Literature

Heterogeneity 1. Clinical: – Population, intervention, outcome (systematic bias) 2. Methodology: – Study design (systematic bias) 3. Statistical – Variability in intervention effects (random error)

Means of addressing heterogeneity • Refrain from performing a meta-analysis • Explore heterogeneity through subgroup analyses or meta-regression • Pre-specify heterogeneity level (fixed-effects) • Perform a random-effects meta-analysis • Change the effect measure • Exclude studies after careful consideration

Fixed effects meta-analyses • Assumes one true treatment effect • Thus, differences across trials can only be due to one source of error (i. e. random error) e. g. TLE secondary to MTS Borenstein et al. , 2007

Random effects meta-analyses • Assumes many treatment effects of which there is a ‘mean’ true effect e. g. epilepsy type

Random effects meta-analyses • Assumes many effects of which there is a ‘mean’ true effect • Two sources of error (random error around the true effect and random error around the mean of true effects) e. g. epilepsy type Borenstein et al. , 2007

The ‘systematic’ approach Conducting the review JAMA Users Guide to the Medical Literature

Conducting the review • Comprehensive search strategy using multiple databases • Some argue that searching MEDLINE, EMBASE, and Cochrane Central is the bare minimum • Additional sources and grey literature: – Trial registries – Reference lists – Personal communications

‘Searching ain’t easy…’ Josephson et al. , Cochrane Database of Systematic Reviews 2014

Databases…just a sampler • • MEDLINE PUBMED EMBASE Cochrane Central TRIP DARE WHO ICTRP Google Scholar

Publication bias Cochrane Handbook

Duplication, duplication • Involvement of two or more reviewers and abstractors ensures reproducibility of the process • Why? Because it is impossible to avoid some degree of subjectivity and, thus, error • The kappa statistic is highly informative (greater IRR suggests more confidence in the process)

Kappa Byrt, Epidemiology, 1996

The ‘systematic’ approach Performing the analyses JAMA Users Guide to the Medical Literature

See the forest for the trees Josephson et al. , Neurology, 2013

Pooling the effect Model effect Assumption methods Measures Fixed effect model Mantel-Haenszel Ratios Peto Odds ratio General variance based Ratios and differences Random effect Der. Simonian and Laird Ratios and differences

Mantel-Haenszel fixed effect Outcome No outcome Exposed a b Unexposed c d M-H weighted average Contingency table

Mantel-Haenszel fixed effect 1 Outcome No outcome 2 Outcome No outcome Exposed 400 200 Exposed 7 8 Unexposed 300 100 Unexposed 2 3 ss. RR= 0. 88 ss. RR= 1. 16 M-H weighted average: ((400*400) + (7*5))/(1000+20) RR= 1 1 2 2 1 2= ((300*600) + (2*15))/(1000+20) 160035/1020 156. 8 = = 0. 89 180030/1020 176. 5

Der. Simonian and Laird Outcome No outcome Exposed a b Unexposed c d DSL Estimate: v = within study variance t = between study variance Contingency table

Was it correct to pool? Josephson et al. , Neurology, 2013

Measuring heterogeneity – Cochran Q Nomenclature: wi = individual study’s weight (1/v) Ti = individual study’s effect size T-bar = mean effect size • Form of chi-square test • p-value of 0. 1 is typically used for significance • Caveat: • Low power = insensitive • High power = too sensitive Cochrane Handbook

Measuring heterogeneity – I 2 statistic • % variability in the effect estimate that is more than chance alone (i. e. due to heterogeneity rather than random error) – – 0 -40% = might not be important 30 -60% = may represent moderate heterogeneity 50 -90% = may represent substantial heterogeneity 75 -100% = considerable heterogeneity Cochrane Handbook

Confidence = ‘GRADE’ the results 1. 2. 3. 4. 5. GRADE Type of evidence (RCT vs. observational) Quality (blinding, allocation, f/u, sparse data, methodology) Consistency of results (within or between studies) Directness (generalisability) Effect size (e. g. >5 or <0. 2 for all studies)

Additional quality scales Jadad: RCTs Newcastle-Ottawa Scale: non-RCTs

Quality summary table (e. g. QUADAS scale) Josephson et al. , Cochrane Database of Systematic Reviews 2014

Summary of findings table Cochrane Handbook

• Required for all studies! • Checklist outlining all essential steps of the review to facilitate quality control • www. prisma-statement. org

Conclusions PICOST Statistical plan (PROSPERO) Performing the analyses JAMA Users Guide to the Medical Literature

Thank you