Value and Limitations of MetaAnalysis in the Era

Value and Limitations of Meta-Analysis in the Era of Evidence-Based Medicine Giuseppe Biondi-Zoccai, MD Division of Cardiology, Department of Internal Medicine, University of Turin, Italy Meta-analysis and Evidence-based medicine Training in Cardiology (METCARDIO), Turin, Italy gbiondizoccai@gmail. com www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

Why should you trust me? Meta analyses or manuscript pertinent to meta analyses that I have co authored since graduation Total = 51 gbiondizoccai@gmail. com www. metcardio. org

Why are meta analysis important: exponential increase in worldwide Pub. Med citations Pub. Med search strategy: ("2001"[PDAT] : "2005"[PDAT]) AND (("systematic"[title/abstract] AND "review"[title/abstract]) OR ("systematic"[title/abstract] AND "overview"[title/abstract]) OR ("meta-analysis"[title/abstract] OR "meta-analyses"[title/abstract])) gbiondizoccai@gmail. com www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

Famous quotes “If I have seen further it is by standing on the shoulders of giants” Isaac Newton “The great advances in science usually result from new tools rather than from new doctrines” Freeman Dyson gbiondizoccai@gmail. com www. metcardio. org

Famous quotes “I like to think of the meta analytic process as similar to being in a helicopter. On the ground individual trees are visible with high resolution. This resolution diminishes as the helicopter rises, and in its place we begin to see patterns not visible from the ground” Ingram Olkin gbiondizoccai@gmail. com www. metcardio. org

Baby steps of meta analysis • 1904 - Karl Pearson (UK): correlation between inoculation of vaccine for typhoid fever and mortality across apparently conflicting studies • 1931 – Leonard Tippet (UK): comparison of differences between and within farming techniques on agricultural yield adjusting for sample size across several studies • 1937 – William Cochran (UK): combination of effect sizes across different studies of medical treatments • 1970 s – Robert Rosenthal and Gene Glass (USA), Archie Cochrane (UK): combination of effect sizes across different studies of, respectively, educational and psychological treatments • 1980 s – exponential development/use of meta analytic methods gbiondizoccai@gmail. com www. metcardio. org

Minimal glossary • Review: viewpoint on a subject quoting different primary authors • Overview: as above • Qualitative review: deliberately avoids a systematic approach • Systematic review: deliberately uses a systematic approach to study search, selection, abstraction, appraisal and pooling • Quantitative review: uses quantitative methods to appraise or synthesize data • Meta analysis: uses specific statistical methods for data pooling and/or exploratory analysis • Individual patient data meta analysis: uses specific stastistical methods for data pooling or exploration exploiting individual patient data → Our goal: systematic review (± meta-analysis) gbiondizoccai@gmail. com www. metcardio. org

Qualitative review gbiondizoccai@gmail. com Tung et al, Ann Intern Med 2006 www. metcardio. org

Systematic review and meta analyses • What is a systematic review? – A systematic appraisal of the methodological quality, clinical relevance and consistency of published evidence on a specific clinical topic in order to provide clear suggestions for a specific healthcare problem • What is a meta-analysis? – A quantitative synthesis that, preserving the identity of individual studies, tries to provide an estimate of the overall effect of an intervention, exposure, or diagnostic strategy gbiondizoccai@gmail. com www. metcardio. org

Systematic review (w/o meta analysis) gbiondizoccai@gmail. com Hackan et al, JAMA 2003 www. metcardio. org

Systematic review and meta analysis gbiondizoccai@gmail. com Agostoni et al, J Am Coll Cardiol 2004 www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

EBM hierarchy of evidence 1. N of 1 randomized controlled trial 2. Systematic reviews of homogeneous randomized trials 3. Single (large) randomized trial 4. Systematic review of homogeneous observational studies addressing patient important outcomes 5. Single observational study addressing patient important outcomes 6. Physiologic studies (eg blood pressure, cardiac output, exercise capacity, bone density, and so forth) 7. Unsystematic clinical observations gbiondizoccai@gmail. com Guyatt and Rennie, Users’ guide to the medical literature, 2002 www. metcardio. org

Parallel hierarchy of scientific studies in cardiovascular medicine Qualitative reviews Systematic reviews Meta analyses from individual studies Meta analyses from individual patient data gbiondizoccai@gmail. com Case reports and series Observational studies Observational controlled studies Randomized controlled trials Multicenter randomized controlled trials Biondi-Zoccai, Ital Heart J 2003 www. metcardio. org

gbiondizoccai@gmail. com www. metcardio. org

gbiondizoccai@gmail. com www. metcardio. org

The Cochrane Collaboration Mission Statement: The Cochrane Collaboration is an world wide organization that aims to help people make well informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions gbiondizoccai@gmail. com www. metcardio. org

The Cochrane Collaboration • • Over 6000 contributors 50 Collaborative Review Groups (CRGs) 12 centers throughout the world 9 fields 11 Methods Groups 1 Consumer Network The Campbell Collaboration (focusing on education/social sciences) gbiondizoccai@gmail. com www. metcardio. org

Cochrane resources • Cochrane Database of Systematic Reviews (CDSR) – contains Cochrane systematic reviews • Database of Abstracts of Reviews of Effectiveness (DARE) – contains abstracts of non Cochrane reviews • Cochrane Central Controlled Trials Register (CENTRAL) – contains titles or abstracts of RCTs from multiple sources • Cochrane Database of Methodology Reviews – contains Cochrane reviews of methods papers • Cochrane Methodology Register (CMR) – contains abstracts of non Cochrane methods papers • Health Technology Assessment Database (HTA) – contains abstracts of HTA papers • NHS Economic Evaluation Database (NHS EED) – contains abstracts of economic analysis papers gbiondizoccai@gmail. com www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

Pros • Application to any clinical research question • Systematic searches for clinical evidence • Explicit and standardized methods for search and selection of evidence sources • Thorough appraisal of the internal validity of primary studies • Quantitative synthesis with increased statistical power • Increased external validity by appraising the effect of an intervention (exposure) across different settings • Test subgroup hypotheses • Explore clinical and statistical heterogeneity Lau et al, Lancet 1998 gbiondizoccai@gmail. com www. metcardio. org

Any application feasible: meta analysis of intervention studies gbiondizoccai@gmail. com Landoni et al, Am J Kidney Dis 2006 www. metcardio. org

Any application feasible: meta analysis of diagnostic studies gbiondizoccai@gmail. com Hamon et al, JACC 2006 www. metcardio. org

Any application feasible: meta analysis of prognostic studies gbiondizoccai@gmail. com www. metcardio. org

Thorough appraisal of internal validity and quality of selected studies Landoni et al, J Cardiothorac Vasc Anesth 2007 gbiondizoccai@gmail. com www. metcardio. org

Increasing statistical power and external validity gbiondizoccai@gmail. com De Luca et al, EHJ 2009 www. metcardio. org

Test subgroup analyses ATC, BMJ 2002 gbiondizoccai@gmail. com www. metcardio. org

Explore statistical and clinical heterogeneity gbiondizoccai@gmail. com Biondi-Zoccai et al, Am Heart J 2005 www. metcardio. org

Explore small study effects gbiondizoccai@gmail. com Abbate et al, J Am Coll Cardiol 2008 www. metcardio. org

Arguably the most important meta analysis ever…. Antman et al, JAMA 1992 gbiondizoccai@gmail. com www. metcardio. org

…showing discrepancies among evidence and experts gbiondizoccai@gmail. com www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

Cons • “Exercise in mega silliness” • “Mixing apples with oranges” • Not original research • Big RCTs definitely better • Pertinent studies might not be found, or may be of low quality or internal validity • Publication and small study bias • Average effect largely unapplicable to individuals • Duplicate efforts may lead to discordant results Lau et al, Lancet 1998 gbiondizoccai@gmail. com www. metcardio. org

What if I mix apples and oranges… Hooper et al, BMJ 2006 gbiondizoccai@gmail. com www. metcardio. org

What if I mix apples and oranges… gbiondizoccai@gmail. com www. metcardio. org

What if only few/low quality studies are found? gbiondizoccai@gmail. com Biondi-Zoccai et al, J Endovasc Ther 2009 www. metcardio. org

What if small positive studies are selectively published? (standard error of log relative risk) Precision 0. 0 P<0. 001 at Egger test P<0. 001 at Peters test 0. 4 0. 8 1. 2 1. 6 0. 01 0. 1 Favours cilostazol 1 Favours control 10 100 Effect (relative risk) gbiondizoccai@gmail. com Biondi-Zoccai et al, Am Heart J 2008 www. metcardio. org

What if meta analyses disagree? Biondi-Zoccai et al, BMJ 2006 gbiondizoccai@gmail. com www. metcardio. org

Appraisal tools: QUOROM Moher et al, Lancet 1999 gbiondizoccai@gmail. com www. metcardio. org

Appraisal tools: Oxman and Guyatt’s Evaluates the internal validity of a review on 9 separate questions for which 3 distinct anwers are eligible (“yes”, “partially/can’t tell”, “no”): 1. Where the search methods used to find evidence stated? 2. Was the search for evidence reasonably comprehensive? 3. Were the criteria for deciding which studies to include in the overview reported 4. Was bias in the selection of studies avoided 5. Were the criteria used for assessing the validity of the included studies reported? 6. Was the validity of all studies referred to in the text assessed using appropriate criteria 7. Were the methods used to combine the findings of the relevant studies reported? 8. Were the findings of the relevant studies combined appropriately relative to the primary question the overview addresses? 9. Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview? Question 10 summarizes the previous ones and, specifically, asks to rate the scientific quality of the review from 1 (being extensively flawed) to 3 (carrying major flaws) to 5 (carrying minor flaws) to 7 (minimally flawed). The developers of the index specify that if the “partially/can’t tell” answer is used one or more times in questions 2, 4, 6, or 8, a review is likely to have minor flaws at best and is difficult to rule out major flaws (ie a score≤ 4). If the “no” option is used on question 2, 4, 6 or 8, the review is likely to have major flaws (ie a score≤ 3). gbiondizoccai@gmail. com Oxman et al, J Clin Epidemiol 1991 www. metcardio. org

Index • How to define meta analyses? Key concepts • What comes first? Scientific hierarchy and The Cochrane Collaboration • Where’s the beef? Strenghts of meta analyses • Any toxic asset? Weaknesses of meta analyses • See one, do one, teach one. Structured approach to systematic reviews gbiondizoccai@gmail. com www. metcardio. org

Algorithm for systematic reviews • Definition of question and hypothetical solution • Prospective design of the systematic review • Data search • Data abstraction and appraisal • Data analysis ± quantitative synthesis FEED-BACK ON HYPOTHESIS • Problem formulation (population, intervention or exposure, comparison, outcome [PICO]) • Result interpretation and dissemination Biondi-Zoccai et al, Ital Heart J 2004 gbiondizoccai@gmail. com www. metcardio. org

Definition of question and prospective design • The clinical question should be clearly stated, being as much explicit as possible • The review should be designed in as much details as possible, and yet with a limited a priori knowledge of the subject Biondi-Zoccai et al, Ital Heart J 2004 gbiondizoccai@gmail. com www. metcardio. org

Problem formulation according to the PICO approach • Population of interest – eg elderly male >2 weeks after myocardial infarction) • Intervention (or exposure) – eg intracoronary infusion of progenitor blood cells • Comparison – eg patients treated with progenitor cells vs standard therapy • Outcome(s) – eg change in echocardiographic left ventricular ejection fraction from discharge to 6 month control Biondi-Zoccai et al, Ital Heart J 2004 gbiondizoccai@gmail. com www. metcardio. org

Data search • After definition of question according to PICO approach, the appropriate key words are used to search several databases • Useful resources: Bio. Med. Central, CENTRAL, clinicaltrials. gov, EMBASE/Scopus, LILACS, and Pub. Med • Conference proceedings • Cross referencing (snowballing) • Contact with experts gbiondizoccai@gmail. com www. metcardio. org

Example of search strategies A simple Pub. Med strategy for clinical studies on percutaneous coronary intervention for left main coronary artery disease: left AND main AND coronary AND stent* NOT case reports [pt] NOT review [pt] NOT editorial [pt] A complex Pub. Med strategy for randomized clinical trials on invasive vs conservative strategies in acute coronary syndromes: (randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized controlled trials[mh] OR random allocation[mh] OR double-blind method[mh] OR single-blind method[mh] OR clinical trial[pt] OR clinical trials[mh] OR (clinical trial[tw] OR ((singl*[tw] OR doubl*[tw] OR trebl*[tw] OR tripl*[tw]) AND (mask*[tw] OR blind[tw])) OR (latin square[tw]) OR placebos[mh] OR placebo*[tw] OR random*[tw] OR research design[mh: noexp] OR comparative study[mh] OR evaluation studies[mh] OR follow-up studies[mh] OR prospective studies[mh] OR cross-over studies[mh] OR control*[tw] OR prospectiv*[tw] OR volunteer*[tw]) NOT (animal[mh] NOT human[mh]) NOT (comment[pt] OR editorial[pt] OR meta-analysis[pt] OR practice-guideline[pt] OR review[pt])) AND ((invasive OR conservative AND (coronary OR unstable angina OR acute coronary syndrome* OR unstable coronary syndrome* OR myocardial infarction))) Biondi-Zoccai et al, Int J Epidemiol 2005 Biondi-Zoccai et al, Am Heart J 2008 Biondi-Zoccai et al, Am Heart J 2005 gbiondizoccai@gmail. com www. metcardio. org

Study selection • 1 st - screening of titles and abstracts • 2 nd – potentially pertinent citations are then retrieved as full reports and appraised according to prespecified and explicit inclusion/exclusion criteria • 3 rd – studies fullfilling both inclusion and exclusion criteria, are then included in the systematic review gbiondizoccai@gmail. com www. metcardio. org

Andreotti et al, Eur Heart J 2005 gbiondizoccai@gmail. com www. metcardio. org

Data abstraction and appraisal • Abstraction of outcomes and moderator variables, possibly on prespecified data form • Appraisal of the internal validity of primary studies (eg the risk of selection, performance, adjudication and attrition bias) • Performed by single vs multiple reviewers, with divergences resolved by consensus (possibly after formal tests for agreement) gbiondizoccai@gmail. com www. metcardio. org

Internal validity of primary studies • Many scales for the quality of included studies have been reported, but none is reliable or robust • The recommended approach is to individually appraise the potential risk of the 4 biases (eg A low, B moderate, C high, D unclear from reported data): – Selection bias (one group is different than the other) – Performance bias (treatment is systematically different) – Adjudication bias (outcome adjudication is selectively different) – Attrition bias (follow up duration or completeness is different) gbiondizoccai@gmail. com www. metcardio. org

Another common classification scheme for bias gbiondizoccai@gmail. com www. metcardio. org

Data synthesis • Quantitative data synthesis is central to the practice of meta analysis, and is based on a major assumptio: individual studies that are going to be pooled are relatively homogeneous, both clinically and statistically, to provide a meaningful central tendency effect estimate gbiondizoccai@gmail. com www. metcardio. org

Effect sizes and p values Forms of research findings suitable to meta analysis: • Central tendency research: – incidence or prevalence rates – mean (standard error) • Pre post contrasts: – changes in continuous or categorical variables • Group contrasts: – experimentally created groups: • comparison of outcomes between experimental and control groups – naturally or non experimentally occurring groups • treatment, prognostic or diagnostic features • Association between variables: – correlation coefficients – regression coefficients gbiondizoccai@gmail. com www. metcardio. org

Effect sizes and p values • The effect size makes meta analysis possible: – it is the “dependent variable” – it standardizes findings across studies such that they can be directly compared • Any standardized index can be an “effect size” as long as it meets the following: – is comparable across studies (generally requires standardization) – represents the magnitude and direction of the relationship of interest – is independent of sample size • We identify p values (for effect) for measuring alpha error for hypothesis testing and corresponding confidence intervals gbiondizoccai@gmail. com www. metcardio. org

Continous variables • Continous variables can be pooled with – Weighted mean differences (WMD), if the same variable is used across studies – Standardized mean differences (SMD), if similar but not identical variables are used – Inverse variance weighting, if only point estimates and standard errors are available gbiondizoccai@gmail. com www. metcardio. org

Relative risks • Relative risks (RR) are defined as the ratio of incidence rates, and are thus used for dichotomic variables) • What is the meaning of RR: – RR=1 means no difference in risk – RR<1 means reduced risk in group 1 vs 2 – RR>1 means increased risk in group 1 vs 2 • RRs are easier to interpret but are less userfriendly from a statistical point of view (RRAvs. B≠ 1/RRBvs. A) and may appear over optimistic gbiondizoccai@gmail. com www. metcardio. org

Odds ratios • Odds ratios (OR) are defined as the ratio of the odds (P/[1 P]) and also used for dichotomic variables • When prevalences are low, they are a good approximation of RR • They behave similarly to RR (OR=1 means no difference in risk, …) • ORs are less easy to interpret but more flexible from a statistical point of view (ORAvs. B=1/ORBvs. A), yet also overoptimistic gbiondizoccai@gmail. com www. metcardio. org

Risk differences and number needed to treat/harm • The risk difference (RD), ie absolute risk difference, is the difference between the incidence of events in the experimental vs control groups • The RD is theoretically the most clinically relevant statistics, but changes too much with disease prevalence • The number to treat (NNT), defined as 1/RD, identifies the number of patients that we need to treat with the experimental therapy to avoid one event* • The NNT is the most clinically meaningful parameter to express the impact of a treatment on a dichotomic outcome (eg death), but has the same limits of RD *Numbers needed to harm (NNH) similarly express the number of patients that we have to treat with the experimental therapy to cause one adverse event gbiondizoccai@gmail. com www. metcardio. org

RR, OR or RD/NNT? OR RR RD/NNT Communication - + ++ Consistency + ++ - Mathematics ++ - - gbiondizoccai@gmail. com www. metcardio. org

Our advice • Both RR and OR can be your first choice statistics for uncommon events • For common events, the OR is clearly less informative than the RR for the busy reader • Complete your analyses by reporting RD and/or NNT for the sake of clarity • Fixed effect methods are quite fine for homogeneous/ consistent data • Random effect methods may be more appropriate for heterogeneous/inconsistent data, but often meta regression (or even refraining from meta analysis at all) might be the best option gbiondizoccai@gmail. com www. metcardio. org

Small study bias • Publication bias (eg the lower likelihood of being published for studies with negative findings, or those originating in non English speaking countries) may bias the results of a meta analysis • Other types of small study bias may undermine the validity of a meta analysis • A number of tests, analogical (eg the funnel plot) or analytical (eg Egger’s or Peter’s) have been proposed to appraise the likelihood of such small study bias Peters et al, JAMA 2006 gbiondizoccai@gmail. com www. metcardio. org

Statistical heterogeneity • Statistical heterogeneity may be suspected by inspecting tables (summary estimates/SE) and forest plots, or analytically • Chi square, Breslow, or Cochran tests are most commonly used • While a 2 tailed p=0. 05 is used for cut off for hypothesis testing of effect, a 2 tailed p=0. 10 is conventionally chosen for heterogeneity gbiondizoccai@gmail. com www. metcardio. org

Statistical inconsistency • Statistical inconsistency (I 2) has been recently introduced to overcome the risk of alpha and beta error of standard tests for statistical heterogeneity • It is computed as [(Q – df)/Q] x 100%, where Q is the chi squared statistic and df is its degrees of freedom • I 2 values of 25% suggest low inconsistency, 50% moderate inconsistency, and 75% severe inconsistency Higgins et al, BMJ 2003 gbiondizoccai@gmail. com www. metcardio. org

Statistical packages • Rev. Man (http: //www. cochrane. org) » For meta analyses of medical interventions • Meta-Test (jlau 1@tufts. edu) FREEWARES! • Easy. MA (http: //www. spc. univ-lyon 1. fr/easyma. net/) • Meta-Di. Sc (http: //www. hrc. es/investigacion/metadisc. html) » For meta analyses of diagnostic tests • • • Fast. Pro NCSS SAS SPSS Stata WEasy. MA gbiondizoccai@gmail. com Not for free • U of Pittsburgh (http: //www. pitt. edu/~super 1/lecture/lec 1171/index. htm) www. metcardio. org

Typical Revman output gbiondizoccai@gmail. com www. metcardio. org

A few references • Biondi Zoccai GGL et al. Parallel hierarchy of scientific studies in cardiovascular medicine. Ital Heart J 2003; 4: 819 20 • Biondi Zoccai GGL et al. Compliance with QUOROM and quality of reporting of overlapping meta analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006; 332: 202 209 • Biondi Zoccai GGL et al. A practical algorithm for systematic reviews in cardiovascular medicine. Ital Heart J 2004; 5: 486 7 • Bucher HC et al. The results of direct and indirect treatment comparisons in meta analysis of randomized controlled trials. J Clin Epidemiol 1997; 50: 683– 9 • Cappelleri JC et al. Large trials vs meta analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332 8 • Clarke M et al, eds. Cochrane reviewers’ handbook 4. 2. 0. (www. cochrane. org/resources/handbook. pdf) • Cooper H et al, eds. The handbook of research synthesis. New York, NY: Russell Sage Foundation, 1994 • Cucherat M et al. Easy. MA: a program for the meta analysis of clinical trials. Comput Methods Programs Biomed 1997; 53: 187 90 • Egger M et al, eds. Systematic reviews in health care: meta analysis in context. 2 nd ed. London: BMJ Publishing Group, 2001 • Glass G. Primary, secondary and meta analysis of research. Educ Res 1976; 5: 3 8 • Glasziou P et al. Systematic reviews in health care. A practical guide. Cambridge: Cambridge University Press, 2001 • Guyatt G et al, eds. Users’ guides to the medical literature. A manual for evidence based clinical practice. Chicago, IL: AMA Press, 2002 • Higgins JPT et al. Measuring inconsistency in meta analyses. BMJ 2003; 327: 557 – 60 • Lau J et al. Summing up evidence: one answer is not always enough. Lancet 1998; 351: 123 7 • Moher D et al. Improving the quality of reports of meta analyses of randomised controlled trials: the QUORUM statement. Lancet 1999; 354: 1896 900 • Petitti DB. Meta analysis, decision analysis, and cost effectiveness analysis: methods for quantitative synthesis in medicine. New York, NY: Oxford University Press, 2000 • Song F et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta analysis. BMJ 2003; 326: 472 • Thompson SG et al. How should meta regression analyses undertaken and interpreted? Stat Med 2002; 21: 1559 73 gbiondizoccai@gmail. com www. metcardio. org

Take home messages • The validity of a meta analysis refers to the soundness of the original studies and the procedures used to combine them (if appropriate) • Dozens of potential validity threats have been identified, and should always be borne in mind • Given its current pivotal role in the hierarchy of clinical evidence, all clinical decision makers should have a working knowledge of how to appraise and/or conduct a systematic review/meta analysis gbiondizoccai@gmail. com www. metcardio. org

Thank you for your attention! For any correspondence: gbiondizoccai@gmail. com For further slides on these topics feel free to visit the metcardio. org website: http: //www. metcardio. org/slides. html gbiondizoccai@gmail. com www. metcardio. org