FixedEffects and Random Effects Models Created for the

Fixed-Effects and Random. Effects Models Created for the third edition of the Users' Guides to the Medical Literature. Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Objective • To explore the differences between fixedeffects and random-effects models used in meta-analyses in greater detail Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Five A’s of EBM Ask Acquire Act Patient Apply Appraise Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Models for Combining Data • In a meta-analysis, results from 2 or more primary studies can be combined statistically using a fixed-effects or a random-effects model • You can consider the differences between the 2 models by looking at • Underlying assumptions • Statistical considerations • How choice of model affects results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

However. . . • This is a controversial area within the field of meta-analysis • Even expert statisticians may disagree with the characterizations on the subsequent slides • Our approach is largely consistent with the Cochrane Collaboration Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Analogy • You enroll 50 teachers in a study of a new math curriculum • For each teacher, you randomize the classes; half of the classes receive the old curriculum and half receive the new one • You then evaluate the effectiveness of the curricula in optimizing student test scores • What is this experiment trying to answer? • There is more than 1 possibility, with more than 1 underlying assumption Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Two Possible Scenarios 1. Among these 50 teachers and no others, what is the effect of the 2 curricula on student examination scores? • Assumption: effect of new vs old curriculum is the same in all teachers 2. Among all teachers who might ever teach this course, of whom these 50 are a random sample, what is the impact of the 2 curricula on examination scores? • Assumption: effect of new vs old curriculum differs among teachers Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Differences Between Scenarios • In terms of the questions: are we interested in the effect of the curricula in these 50 teachers or the effect in all teachers? • In terms of assumptions: the relative effect of the old and new curricula is the same in each of these 50 teachers vs different across teachers Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Completing the Analogy • Substitute “studies” for teachers and “therapies” for curricula and you have the questions and assumptions for fixedeffects (question 1) and random-effects (question 2) models Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Fixed-Effects Methods • It is important to know that there are different statistical approaches for fixedeffects models • Inverse variance • Mantel-Haenszel method • Peto odds ratio • No one knows the best approach • Rarely, choice of method may yield noticeable differences in results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Random-Effects Methods • Multiple methods also exist for randomeffects models that differ in how they approximate between-study variability • Most commonly used is Der. Simonian and Laird method • Random-effects model methods also can weight studies using either the inverse variance method or the Mantel-Haenszel method Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Fixed-Effects Model? • A fixed-effects model assumes that there is a single true value underlying all results of studies included in the meta-analysis • If all studies that address the same question were infinitely large and completely free of bias, they would yield identical estimates of effect • Observed estimates of effect differ from one another only because of random error Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Fixed-Effects Model? • A fixed-effects model does not consider between-study variability in results; the error term comes only from within-study variation • Called study variance • This model aims to estimate this commontruth effect and the uncertainty about it Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Random-Effects Model? • A random-effects model assumes that the studies included are a random sample of a population of studies that address the question posed in the meta-analysis • Because there are inevitably differences in the patients, interventions, and outcomes among studies, each study estimates a different underlying true effect • These effects will have a normal distribution Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

What Is a Random-Effects Model? • The pooled estimate in a random-effects model is not a single effect of the intervention • Rather, it is the mean effect across the different populations, interventions, and methods of outcome evaluation • This model takes into account both withinstudy variability and between-study variability Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Comparison of Models Fixed-Effects Models Random-Effects Models Conceptual considerations Estimates effect in this sample of studies Assumes effects are the same in all studies Estimates effect in a population of studies from which the available studies are a random sample Assumes effects differ across studies and the pooled estimate is the mean effect Statistical considerations Variance is only derived from within-study variance Variance is derived from both within -study and between-study variances Practical considerations Narrow CI Large studies have much more weight than small studies Wider CI Large studies have more weight than small studies, but the gradient is smaller than in fixed-effects models Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Differences in Results • Sometimes, results are similar from study to study • For statistical pooling, this will mean that between-study variability can be fully explained by chance • Between-study variance estimated to be 0 • Corresponds to an I 2 of 0% • Under these circumstances, fixed-effects and random-effects models will give identical results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Differences in Results • In approximately 40% of Cochrane metaanalyses of binary outcomes of RCTs, results are sufficiently similar across trials that variability can be explained by chance and I 2 is 0 • This situation occurs in a smaller percentage of meta-analyses of epidemiologic studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Differences in Results • In another 40% or so of meta-analyses of RCTs, the estimated between-study variance is not 0 but not large • Both fixed-effects and random-effects models provide quite similar results • In the final 20%, the between-study variability is large, and fixed-effects and random-effects models yield disparate results that may have important implications Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Effect of Model Choice on Precision • Because estimation of variance under the random-effects model includes betweenstudy variability, when results vary across studies, the CI of the combined estimate will be wider • The random-effects model generally produces a more conservative assessment of the precision of the summary estimate than the fixed-effects model Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Hypothetical Example of Significant Variability Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Hypothetical Example of Significant Variability Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Effect of Model Choice on Point Estimate • In both models, larger studies have larger weight • A random-effects model gives smaller studies proportionally greater weight in the summary estimate • Direction and magnitude of summary estimate are influenced relatively more by smaller studies • These models thus generate summary estimates closer to null result than fixed-effects estimates if smaller study results are closer to null result than those from larger studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Effect of Model Choice on Point Estimate • If smaller studies are farther from the null result than larger studies, a randomeffects model will produce larger estimates of beneficial or harmful effects than will a fixed-effects model • Summary estimate derived from the random-effects model may be more susceptible to overestimates from small studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Hypothetical Example of Significant Variability and Small Studies That Have Different Estimates Than Large Studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Four Guidelines to Follow • Which model to believe? • Statisticians and clinical trialists can be passionate about fixed-effects and randomeffects models, and viewpoints differ • The following 4 guidelines may help you decide which model to believe when results differ between the 2 models Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Guideline 1 • If there is little variability among studies, fixed-effects and random-effects point estimates and CIs will vary little Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Guideline 2 • Uncertainty about accuracy/applicability of a point estimate increases with increasing variability in study results • • Random-effects model captures this uncertainty with wider CIs It is also conceptually appealing • • We are interested not just in available studies but in applying them to a wider population It is also likely that true effects differ across populations and thus across studies Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Guideline 3 • Fixed-effects model is preferable when one study is much larger and more trustworthy than one or more smaller studies that address the same question and yield quite different results Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Guideline 4 • Fixed-effects model also may be preferable when the number of studies included in a meta-analysis is very small (< 5), leading to concern about inaccurate estimation of between-study variance Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects models: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 1 • You are a surgeon evaluating a patient presenting with a localized renal tumor • You have 2 treatment options: partial or radical nephrectomy • You are interested in knowing the relative impact of each of the 2 procedures on cancer-specific mortality • A systematic review and meta-analysis compared the 2 interventions Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Meta-analysis Comparing Partial and Radical Nephrectomy on Cancer-Specific Mortality Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 1 • The study authors presented results using both models • Under the fixed-effects model, results are statistically significant in favor of partial nephrectomy • HR, 0. 71; 95% CI, 0. 59 -0. 85; P <. 01 • However, using the random-effects model, results are no longer significant • HR, 0. 79; 95% CI, 0. 57 -1. 11; P =. 17 Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 1 • This analysis was associated with substantial heterogeneity • The extreme differences in results substantially reduce confidence in the summary estimate of effect • This reduced confidence is reflected in the wider CI of the random-effects model, which in this instance is more appropriate Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 2 • You are evaluating a patient presenting with myocardial infarction and recall that intravenous magnesium has been used in this setting • You find a systematic review and metaanalysis that evaluated the effect of magnesium on mortality for patients with myocardial infarction Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Meta-analysis Comparing Magnesium to Control Therapy in Patients With Acute Myocardial Infarction Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 2 • The meta-analysis included 22 trials, and the analysis was associated with moderate heterogeneity (I 2 = 64%) • Most of these trials were relatively small, but 2 were not • Many of the small trials found an apparently significant reduction in mortality • The 2 largest trials, however, found no benefit Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Example 2 • In this case, we are inclined to believe the results of the 2 large studies • The random-effects models results are therefore misleading • The figure also shows the relative weight of each trial • For example, the largest trial (ISIS-4) has a relative weight of almost 75% under the fixed -effects model, but only 18% under the random-effects model Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Outline • Introduction • Fixed-effects vs random-effects: an analogy • Models for combining data • Practical considerations • When results differ between the 2 models • Examples of differences in point estimates and confidence intervals • Conclusion Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Conclusion • There is no right answer as to which model is best • With the knowledge you have of the differences between the 2 models, you can make your own choice • It may make little difference which model data analysts choose, but understanding the implications of their choice will help you make sense of situations in which large variability in study results exists Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Terms of Use: Users Guides to the Medical Literature Education Guides Power. Point Usage Guidelines JAMAevidence users may display, download, or print out Power. Point slides and images associated with the site for personal and educational use only. Educational use refers to classroom teaching, lectures, presentations, rounds, and other instructional activities, such as displaying, linking to, downloading, printing, and making and distributing multiple copies of said isolated materials in both print and electronic format. Users will only display, distribute, or otherwise make such Power. Point slides and images from the applicable JAMAevidence materials available to students or other persons attending in-person presentations, lectures, rounds, or other similar instructional activities presented or given by User. Commercial use of the Power. Point slides and images are not permitted under this agreement. Users may modify the content of downloaded Power. Point slides only for educational (non-commercial) use; however, the source and attribution may not be modified. Users may not otherwise copy, print, transmit, rent, lend, sell, or modify any images from JAMAevidence or modify or remove any proprietary notices contained therein, or create derivative works based on materials therefrom. They also may not disseminate any portion of the applicable JAMAevidence site subscribed to hereunder through electronic means except as outlined above, including mail lists or electronic bulletin boards. Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.

Created by Gordon Guyatt, MD, Kate Pezalla, MA, and Annette Flanagin, RN, MA Users’ Guides to the Medical Literature JAMA | Centre for Health Evidence | The Mc. Graw-Hill Companies, Inc. Copyright © American Medical Association. All rights reserved.