Appraising Evidence From a Systematic Review and Metaanalysis

Appraising Evidence From a Systematic Review and Meta-analysis Updated for the third edition of the Users' Guides to the Medical Literature. Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Objectives • Clinical scenario • Understanding the process of a systematic review and meta-analysis • User’s Guides • Evaluating the credibility of the systematic review process • Summary • Clinical scenario resolution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Objectives • Be able to • Understand the process of a systematic review and meta-analysis • Apply the Users’ Guides for systematic reviews and meta-analyses • Identify threats to the credibility of the systematic review process Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Five A’s of EBM Ask Acquire Act Patient Apply Appraise Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Objectives • Clinical scenario • Understanding the process of a systematic review and meta-analysis • User’s Guides • Evaluating the credibility of the systematic review process • Summary • Clinical scenario resolution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Clinical Scenario • You receive request for consultation from a general surgeon • Patient is 66 -year-old man undergoing hip surgery in 2 days; he is a smoker and has • Type 2 diabetes • Hypertension • No history of heart disease • Blood pressure of 135/80 mm Hg Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is the Question? • Should patients undergoing noncardiac surgery receive β-blockers? • Question of therapy Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Finding the Evidence • A large amount of literature exists on this controversial topic • You need an accurate overview of current best evidence • You are hoping to find a recent systematic review and meta-analysis of RCTs that deal with this topic Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Finding the Evidence • You start your search with the free federated search engine ACCESSSS (http: //plus. mcmaster. ca/accessss) • Search terms • Beta blockers • Perioperative • Mortality Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Finding the Evidence • Summaries at the top of your search output have not been recently updated; you look further down to check preappraised research • You identify a more recently published systematic review and meta-analysis • It addresses your question and was highly rated for relevance and newsworthiness • You download the full text Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Objectives • Clinical scenario • Understanding the process of a systematic review and meta-analysis • User’s Guides • Evaluating the credibility of the systematic review process • Summary • Clinical scenario resolution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Systematic Review? • Summary of research that addresses a focused clinical question in a systematic, reproducible manner • Can provide estimates of therapeutic efficacy, prognosis, and diagnostic test accuracy • Can summarize the evidence for questions of “how” and “why” addressed by qualitative research studies • Often accompanied by a meta-analysis Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Meta-analysis? • A statistical pooling or aggregation of results from different studies • Pooling studies increases precision (ie, narrows the measures of error or variance, such as CIs) • Single best effect estimate generated facilitates clinical decision making Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Overlap of Designs Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Why Seek Systematic Reviews? 1. Single studies are likely to be unrepresentative of total body of evidence; results may therefore be misleading 2. Collecting and appraising multiple studies takes time you probably do not have 3. Systematic reviews are often accompanied by meta-analyses to provide best estimate of effect Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Why Seek Systematic Reviews? 4. A well-performed systematic review will likely provide all of the relevant evidence with an assessment of best estimates of effect and the confidence they warrant 5. Systematic reviews include a greater range of patients than any single study, potentially enhancing applicability of results to the patient before you Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Synopsis of the Process Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Defining the Question • Process of a systematic review and metaanalysis begins with defining the question, or specifying eligibility criteria for deciding which studies to include • Criteria define the population, exposures or interventions, and outcomes of interest • A systematic review also may restrict studies to those that minimize risk of bias, such as only including RCTs Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Acquiring the Evidence • Reviewers conduct a comprehensive literature search in all relevant medical databases • Selection criteria are applied to titles and abstracts, narrowing down the large number of results from the initial search • Reviewers can then apply the selection criteria once again, this time to the complete reports Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Appraising the Evidence • After culling the results of the literature search, reviewers assess the risk of bias of the individual studies and abstract data from each study • Data can then be analyzed and the results summarized, including a meta-analysis, if appropriate Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Appraising the Evidence • Meta-analyses provide pooled estimates of the effect on each of the outcomes of interest, along with the associated CIs • May also explore heterogeneity by examining effect estimates across the studies to explain differences in results • If based on previously specified hypotheses about possible differences in patients, interventions, or outcomes, such explorations become more credible Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Objectives • Clinical scenario • Understanding the process of a systematic review and meta-analysis • User’s Guides • Evaluating the credibility of the systematic review process • Summary • Clinical scenario resolution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Users’ Guides • Did the review explicitly address a sensible clinical question? • Was the search for relevant studies exhaustive? • Was the risk of bias of the primary studies assessed? • Did the review address possible explanations of between-study differences in results? • Did the review present results that are ready for clinical application? • Were selection and assessments of studies reproducible? • Did the review address confidence in effect estimates? Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is Credibility? • Credibility refers to the extent to which the design and conduct of the review are likely to have protected against misleading results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Was the Process Credible? • Did the review explicitly address a sensible clinical question? • When deciding whether the question posed in a meta-analysis is sensible, ask yourself whether the underlying biology is such that you would anticipate more or less the same treatment effect across the range of patients included Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Evaluating Eligibility Criteria • Were eligibility criteria for inclusion in the systematic review appropriate? • Are results likely to be similar across the range of included patients? • Are results likely to be similar across the range of studied interventions or exposures? • Are results likely to be similar across the range of ways the outcome was measured? Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • The systematic review you found included 11 trials that enrolled 10 000 patients • All having noncardiac surgery • Randomized to β-blockers or control group • The trials addressed the main outcomes of interest • Death, nonfatal myocardial infarction, nonfatal stroke • β-Blocker, dose, timing, and duration of administration varied across trials Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Assessing the Search • Systematic reviews are at risk of presenting misleading results if they fail to secure a representative sample of eligible studies • Searching a single database is insufficient and can lead to missing important studies • Searching multiple databases is recommended for most clinical questions Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • The authors of the systematic review you found searched MEDLINE, EMBASE, CINAHL, CENTRAL, and other trial registries and databases • Also checked reference lists for more sources • Did not restrict search to particular language or location Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Avoiding Bias • Reporting bias occurs in a number of forms • Publication bias is failure to report or publish studies with negative results; this may cause misleading results of systematic reviews that did not include unpublished studies • Selective outcome reporting bias occurs when investigators measure a number of outcomes but report only those that favor the experimental intervention or those that favor the intervention most strongly Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • In this meta-analysis, the authors conducted a sensitivity analysis that excluded studies with a higher risk of bias • Authors also tested for publication bias • They did not report that they had contacted authors of primary studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Risk of Bias in Primary Studies • A highly credible review will assess the risk of bias of the primary studies it includes • Differences in study methods might explain important differences among results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • The study authors used the Cochrane Collaboration risk of bias assessment methods • Explicitly described risk of bias of each trial by reporting on • Adequacy of generation of allocation sequence • Allocation concealment • Blinding of participants, personnel, and outcome assessors Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Between-Study Differences • Studies included in a systematic review are unlikely to show identical results • Authors should attempt to explain the reasons for variability in results • Chance is always a possibility, or could be due to differences in • Characteristics of patients enrolled • Way the outcome was assessed • Risk of bias Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Applicability of Results • Important to know absolute effect of the intervention • Absolute benefit (or harm) that patients will achieve with therapy depends on their baseline risk • However, relative effects tend to be much more consistent across studies • Reason why meta-analyses of binary outcomes should usually include the relative risk, odds ratio, or hazard ratio and CIs Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Applicability of Results • How then do we determine the absolute effects in which we are interested? • Best way is to obtain an estimate of patients’ baseline risk and then use the relative risk to estimate that patient’s risk difference Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Reproducibility • Authors of systematic reviews decide which studies to include, extent of the risk of bias, and what data to abstract • These decisions require judgment by the reviewers and are subject to both mistakes and bias • Having 2 or more people make each decision can help guard against errors; agreement beyond chance can help with risk of bias Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • In the systematic review you located, 2 independent reviewers assessed trial eligibility and select studies • Third review author resolved disagreements Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Agreement Beyond Chance • If there is good agreement beyond chance among reviewers, we can have more confidence in results • Reviewers often quantify the level of agreement on study selection and appraisal of the risk of bias • Chance-corrected agreement is proportion of possible agreement achieved beyond what one would expect by chance alone • Often measured with κ statistic Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Return to the Clinical Scenario • The study authors did not quantitatively report the agreement level among reviewers, a feature you would have preferred to know Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Confidence in Effect Estimates • Ideally, systematic review authors will explicitly address risk of bias that can diminish confidence in estimates as well as imprecision (ie, wide CIs) and inconsistency (ie, large variability in results from study to study) • If authors do no make explicit assessments themselves, they should at least provide information to help you make your own assessment Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

EQUATOR Guidelines • The EQUATOR Network offers guidelines for reporting systematic review and metaanalyses, found here in the PRISMA Statement: http: //www. equatornetwork. org/reporting-guidelines/prisma Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Objectives • Clinical scenario • Definitions and process • User’s Guides • Credibility of the systematic review process • Summary • Clinical scenario resolution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Summary • A well-performed systematic review with meta-analysis is the best place to start when seeking evidence about the effects of health interventions • When appraising the evidence, ask • Did the authors find all important studies? • Did the synthesis weigh for quality? • Is heterogeneity explained? Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Clinical Scenario Resolution • In assessing the credibility of the systematic review and meta-analysis that you found, you had a couple of areas of concern • Study did not quantitatively report the agreement level among the 3 reviewers • Authors did not report if they had contacted the authors of the primary studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Clinical Scenario Resolution • Overall, you conclude that the credibility of the methods of this systematic review and meta-analysis is moderate to high • You decide to examine the estimates of effect and the associated confidence in these effects Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Terms of Use: Users Guides to the Medical Literature Education Guides Power. Point Usage Guidelines JAMAevidence users may display, download, or print out Power. Point slides and images associated with the site for personal and educational use only. Educational use refers to classroom teaching, lectures, presentations, rounds, and other instructional activities, such as displaying, linking to, downloading, printing, and making and distributing multiple copies of said isolated materials in both print and electronic format. Users will only display, distribute, or otherwise make such Power. Point slides and images from the applicable JAMAevidence materials available to students or other persons attending in-person presentations, lectures, rounds, or other similar instructional activities presented or given by User. Commercial use of the Power. Point slides and images are not permitted under this agreement. Users may modify the content of downloaded Power. Point slides only for educational (non-commercial) use; however, the source and attribution may not be modified. Users may not otherwise copy, print, transmit, rent, lend, sell, or modify any images from JAMAevidence or modify or remove any proprietary notices contained therein, or create derivative works based on materials therefrom. They also may not disseminate any portion of the applicable JAMAevidence site subscribed to hereunder through electronic means except as outlined above, including mail lists or electronic bulletin boards. Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Created by Gordon Guyatt, MD, Kate Pezalla, MA, and Annette Flanagin, RN, MA Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.