Systematic Reviews of the Literature and Metaanalyses problems

Systematic Reviews of the Literature and Meta-analyses: …. problems or panacea? Daren K. Heyland, MD, FRCPC, MSc Queen’s University, Kingston, Ontario

Ø Updated Jan 2009 Ø Summarizes >200 trials studying 21283 patients Ø 34 topics 17 recommendations www. criticalcarenutrition. com

Clinical Practice Guidelines evidence + practice guidelines integration of values Validity Homogeneity Safety Feasibility Cost

In Search of Truth. . . …Does it work? Ø Begins with a hypothesis or question Ø Ø Ø Does Drug X reduce the incidence of problem Y in patients with condition Z Application of experimental or observational methods to determine the answer Results of our observations leads to conclusions that are correct (truth) or incorrect (due to bias or chance)

Levels of Evidence Systematic reviews Ø RCT’s Ø Cohort Studies Ø Case Control Ø Case Series Ø less bias/strong inferences more bias/weaker inferences

8 7 6 00 u 2 4 3 2 1 0 5 00 u 2 00 u 2 8 9 00 u 2 99 u 1 7 6 99 u 1 4 3 5 99 u 1 1 2 99 u 1 99 0 9 8 u. RCT u 1 99 u 1 98 u 1 7 6 5 98 u 1 3 6 98 u 1 97 u 1 u 250 Average Patient Population Size per Year u 200 u 150 u 100 u 50 u 0 198 RCT’s Reviewed in Critical Care Nutrition Guidelines

PLOS 2008; 5: e 4

Learning Objectives u Will be able to appraise and incorporate results of systematic reviews into clinical decision making. u understand the role of systematic reviews in research and policy settings. u List the strengths and weakness of meta-analyses

Overview u u u Definition and Classification Usefulness Methodological Quality Making Inferences Conclusions

Systematic Review… u u Form of scientific investigation to assess the effectiveness of healthcare interventions Integrative research Subjects= original or primary studies Employs methods that limit bias and reduce random error

Systematic Reviews and Meta-analysis Narrative Reviews Systematic Reviews Meta-analysis

Number of Systematic Reviews Published

The Frailties of Narrative Reviews u u If the original studies of thrombolytics therapies had been subject to a systematic review, the treatment effect would have been apparent in the 1970 s instead of 1980 s. Narrative reviews omitted effective therapies and endorsed ineffective therapies. Antman JAMA 1992; 268; 240 and Lau NEJM 1992; 327: 248

Clinical Decision Making and Systematic Reviews u Case Scenario u u 77 y. o. male with presumptive Dx of Urosepsis PMHX: MI, Prostate BMI 21 After initial resuscitation u u u Fi. O 2 = 100%, PO 2 = 55 MAP = 65, CVP 13, levophed 20 mcg/kgk/min rising Cr, 20 ml of urine, acidemic High NG drainage Going to start on EN but not likely to tolerate Role for early supplemental PN?

Clinical Decision Making and Systematic Reviews u Problem u 100 s of citations across scores of journals published over the last 20 years In diverse patient populations or diverse settings with variable or inconsistent results! How do you make sense of this all?

Impact of Caloric Debt Adequacy of EN § Caloric debt associated with: § § Longer ICU stay Days on mechanical ventilation § Complications § Mortality Rubinson CCM 2004; Villet Clin Nutr 2005; Dvir Clin Nutr 2006; Petros Clin Nutr 2006

2007 International Nutrition Practice Survey u u u Point prevalence survey of nutrition practices in ICU’s around the world conducted Jan. 27, 2007 Enrolled 2772 patients from 158 ICU’s over 5 continents Included ventilated adult patients who remained in ICU >72 hours

Hypothesis u u u There is a relationship between amount of energy and protein received and clinical outcomes (mortality and # of days on ventilator) The relationship is influenced by nutritional risk BMI is used to define chronic nutritional risk

What Study Patients Actually Rec’d u Average Calories in all groups: u 1034 kcals and 47 gm of protein Result: u Average caloric deficit in Lean Pts: u u 7500 kcal/10 days Average caloric deficit in Severely Obese: u 12000 kcal/10 days

Relationship Between Increased Calories and 60 day Mortality BMI Group Odds Ratio Overall 0. 76 <20 95% Confidence Limits P-value 0. 61 0. 95 0. 014 0. 52 0. 29 0. 95 0. 033 20 -<25 0. 62 0. 44 0. 88 0. 007 25 -<30 1. 05 0. 75 1. 49 0. 768 30 -<35 1. 04 0. 64 1. 68 0. 889 35 -<40 0. 36 0. 16 0. 80 0. 012 >=40 0. 63 0. 32 1. 24 0. 180 Legend: Odds of 60 -day Mortality per 1000 kcals received per day adjusting for nutrition days, BMI, age, admission category, admission diagnosis and APACHE II score.

u. RESULTS: WHO IS AT RISK?

RCT Level of Evidence that More EN= Improved Outcomes § RCTs of aggressive feeding protocols § § Results in better protein-energy intake Associated with reduced complications and improved survival u. Taylor § et al Crit Care Med 1999; Martin CMAJ 2004 Meta-analysis of Early vs Delayed EN § § Reduced infections: RR 0. 76 (. 59, 0. 98), p=0. 04 Reduced Mortality: RR 0. 68 (0. 46, 1. 01) p=0. 06 www. criticalcarenutrition. com

More is Better! Our Field of Dream If you feed them (better!) They will leave (sooner!)

ICU patients are not all created equal…should we expect the impact of nutrition therapy to be the same across all patients? b

What if you can’t provide adequate nutrition enterally? … to TPN or not to TPN, that is the question!

Current practice in nutritional support in septic patients: Results of national, prospective multicenter German Study u u u Point prevalence study 454 ICUs from 310 hospitals in Germany 399 patients septic patients included u u Median APACHE II 26 68% had no GI pathology 46% in shock Overall mortality 55. 2% u. Elke CCM 2008; 36: 1762

Current practice in nutritional support in septic patients: Results of national, prospective multicenter German Study P=0. 005 u u u Point prevalence study 454 ICUs from 310 hospitals in Germany 399 patients septic patients included u u Median APACHE II 26 68% had no GI pathology 46% in shock Overall mortality 55. 2% Multivariate analysis: PN independent predictor for mortality (OR 2. 09, 95% CI 1. 29 -3. 37)

Early Supplemental PN is Associated with Increased Infection in Critically Ill Trauma Patients u u Retrospective, multicenter, cohort study of 597 severely injured patients Compared those that rec’d PN within 7 to those who did not. Also compared early PN group to subgroup of ‘EN tolerant’ (tolerated 1000 kcal any day during first week) Adjusted for differences in key baseline demographics Sena J Am Coll Surg 2008; 207: 459

Early Supplemental PN is Associated with Increased Infection in Critically Ill Trauma Patients No Early PN Odds Ratio P value Nosocomial Infections 27% 56% 2. 1 (1. 3 -3. 5) P=0. 003 Late ARDS 1% 8% 3. 4 (1. 0 -11. 0) P=0. 04 Death 8% 23% 1. 5 (0. 8 -3. 0) P=0. 24 Nosocomial Infections 42% 69% 2. 5 (1. 1 -5. 9) P=0. 03 Late ARDS 2% 9% 5. 4 (1. 1 -27. 4) P=0. 04 Death 8% 19% 2. 7 (0. 8 -9. 3) P=0. 10 Overall Adjusted EN tolerant analysis Differences not due to differences in glycemic control

Prospective Studies of Supplemental PN Effect on Mortality www. criticalcarenutrition. com

What if you can’t provide adequate nutrition enterally? … to TPN or not to TPN, that is the question! Maximize EN delivery prior to initiating PN

Use of Supplemental PN in Sepsis? u u Results of meta-analysis Results of single RCTs of Septic Patients Results of observational studies Consideration of Individual Patient Characteristics

Using Systematic Reviews in Clinical Practice u u Summarizes large body of knowledge Answers specific clinical question Less likely to be biased than narrative reviews More accurate and precise estimate of treatment effect

Using Systematic Reviews in Research Setting u Research Question: u u What is the effect of Glutamine and Antioxidant supplementation on survival in critically ill patients? Methods: u A meta-analysis

Effect of Glutamine in Critically Ill: A Systematic Review of the Literature u Comprehensive search u Selection criteria u Randomized u Surgical or critically ill adults u Glutamine (EN or PN) vs. placebo u Clinically important outcomes 20 RCT’s

Effect of Glutamine: A Systematic Review of the Literature Mortality Updated Jan 2009, see www. criticalcarenutrition. com

Effect of Glutamine: A Systematic Review of the Literature Infectious Complications Updated Jan 2009, see www. criticalcarenutrition. com

Effect of Glutamine: A Systematic Review of the Literature Hospital Length of Stay Updated Jan 2009, see www. criticalcarenutrition. com

Results of Subgroup Analysis EN (n=9) PN (n=17) Mortality Infection 0. 81 (0. 48 -1. 34) P=0. 41 0. 71 (0. 55 -0. 92) P=0. 008 0. 83 (0. 64 -1. 08) P=0. 16 0. 76(0. 62 -0. 93) P=0. 008 PN>>>EN?

REducing Deaths from OXidative Stress: The REDOXS study Factorial 2 x 2 design 1200 ICU patients Evidence of organ failure Fed enterally glutamine R antioxidants R Concealed placebo Stratified by site placebo antioxidants R placebo

Using Systematic Reviews in Research Setting u u u Summarizes what is known; identifies gaps Background of grant proposals Generates hypotheses Estimate of treatment effect N Subgroup analysis

Using Systematic Reviews in Policy Making As an ICU, should you make an argininesupplemented diet available for general use in your institution?

Meta-analyses of Arginine-supplemented Diets o 22 RCTs of IEDs Ø Ø All arginine-containing IED, not just IMPACT/IMMUNAID Non english, more recently published studies Excluded duplicates Excluded single agents Heyland JAMA 2001; 286: 944

Overall Effect on Mortality u. RR 1. 10 (0. 93 -1. 31)

Overall Effect on Complications u. RR 0. 66 (0. 54 -0. 80)

1. 18 (0. 88, 1. 58)

Effect of Arginine-supplemented Diets Mortality in the Critically Ill Patient Updated Jan 2009, see www. criticalcarenutrition. com

Effect of Arginine-supplemented Diets in the Critically Ill Patient Infectious Complications Updated Jan 2009, see www. criticalcarenutrition. com

Effect of Arginine-supplemented Diets in the Critically Ill Patient Hospital Length of Stay Updated Jan 2009, see www. criticalcarenutrition. com

Using Systematic Reviews in Policy Making u u u Greatest generalizability Consistent with perspective of policy makers Related to other forms of integrative research

Assessing the Validity of Systematic Reviews Validity= fxn { inputs, process, results }

Assessing the Validity of Systematic Reviews u Inputs u selection of studies u u clinical homogeneity explicit, reproducible criteria methodological quality of studies outdated/unmeasured co-interventions

Assessing the Validity of Systematic Reviews u Process u comprehensive search strategy u u u publication/timing bias data excess language bias judgements about inclusion explicit/reproducible data abstraction reproducible

Assessing the Validity of Systematic Reviews u Results u u u few studies few clinical endpoints statistical heterogeneity

Methdological Quality of Meta-analyses lots of bias weak inferences little bias strong inferences Strong clinical recommendations

Making Inferences from a Meta. Analysis of RCT’s Weaker Inferences u u u Small number of trials Weak trial methodology Outdated/unmeasured co-interventions Surrogate endpoints Statistical heterogeneity Fixed effects model Stronger Inferences u u u Large number of trials Strong trial methodology Current/documented cointerventions Clinically important endpoints Statistical homogeneity Random effects model

Meta-analysis vs. Large RCT’s “…if no subsequent randomized, clinical trial, the meta-analysis would have led to the adoption of an ineffective treatment in 32% cases and rejection of useful treatment in 33% cases. ” Le. Lorier NEJM 1997; 337: 536 “I still prefer conventional narrative reviews … Editorial, NEJM

Meta-analysis vs. Large RCT’s RCT #2 RCT #1 RCT #3 RCT #5 RCT #4

Meta-analysis vs. Large RCT’s u u u Argument is with Meta-analysis, not the concept of systematic reviews Assumes the latest single large trial is the GOLD standard Assumes RCT and Meta-analysis are measuring the same thing Differences in Generalizability Bias exists in both TOOLS.

Resolving Discrepancies Between a Metaanalysis and a Subsequent Large RCT Ø Ø Recent meta-analysis found calcuim supplementation to be effective in preventing preeclampsia Large RCT found no risk reduction in health nulliparous women Exploration of heterogeneity across studies Stratify for high and low baseline risk JAMA 1999; 282: 664

Resolving Discrepancies Between a Metaanalysis and a Subsequent Large RCT JAMA 1999; 282: 664

JAMA 2008; 300: 933

Role of Systematic Reviews in Medical Education u u u Good source of medical knowledge Promotes EBM practices Helps locate original articles Facilitates critical appraisal of original research Considered a scholarly research activity

Conclusions u u Important tool to determine the effectiveness of therapeutic interventions Need to understand the strengths, weaknesses and limitations Useful in clinical and policy decision making and research setting Encourage use of and generation of systematic reviews amongst learners.