Systemic Review and Meta Analysis in Cancer Epidemiology

Systemic Review and Meta. Analysis in Cancer Epidemiology Chun Rebecca Chao, Ph. D. Kaiser Permanente Southern California Department of Research and Evaluation 1

Overview n Introduction of systematic review and meta- analysis n Methods of systemic review and meta- analysis 2

What is Systemic Review? n A review that has been prepared using a systemic approach to minimize biases and random errors (which is documented in a materials and methods section)*. n Review in which there is a comprehensive search for ALL relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method. n Systemic review vs. Narrative review n Narrative Review: traditional expert review, subjective, no formal rules in selecting studies, no standard statistical methods for combining studies. *Chalmers and Altman. Systemic Reviews, BMJ Publishing Group, 1995. 3

What is Meta-Analysis? n A statistical analysis of the results from independent studies, which generally aims to produce a single estimate of a treatment effect. n A systemic review may or may not include a metaanalysis. n It is always appropriate and desirable to systemically review a body of data, but it is sometimes inappropriate to statistically pool results from different studies. 4

The Need for Systemic Review n Health Care Professional: number of biomedical publication has been increasing rapidly. n Guideline and Policy Makers: Evidence based medicine is the trend for patient care, clinical guideline development and policy making. n Researchers: future research direction can be guided by systemic reviews. n Consumers 5

Strength of Evidence Concerning Efficacy of Treatment n n n n n Case report Case series without controls Series with literature controls Series with historical controls Case control studies Cohort studies Randomized controlled trials (RCTs) SR/Meta-analysis of RCTs Prospective meta-analysis of RCTs with individual data 6

Strengths of Systemic Review n To provide a more objective appraisal of the evidence § Contribute to resolve uncertainty when original research, reviews and editorials disagree. § Meta-analysis of the BCG vaccine trials 7 Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ

Strengths of Systemic Review (Cont. ) n To provide a more objective appraisal of the evidence § Contribute to resolve uncertainty when original research, reviews and editorials disagree. n To reduce the probability of false negative results n To explore treatment effects in subgroups of patients n To explore and explain heterogeneity between study results n To guide the direction of future studies n To understand current gap and limitation in the literature n To generate new research questions to be addressed. 8

Advantages of Use of Meta-Analysis to Combine Studies n When individual trials or studies are too small to give reliable answers n When large trials or studies are impractical or impossible n Potentially lead to more timely introduction of effective treatment n When there have been many trials or studies showing small effects may be important 9

Number of Publications about Meta. Analysis, 1985 -2005 Results from MEDLINE search using Me. SH and text word “meta-analysis” 10

Potential Limitation of Conducting Systemic Review n Bias can be introduced in reviews in several ways. n Problems associated with design or reporting of original studies n Limitations of using published data n Publication bias 11

Publication Bias – A real threat for systemic review n Studies with significant results are n more likely to be published n More likely to be published without delay (lag time) n More likely to be published in English n More likely to be cited n More likely to be published more than once n Outcome reporting bias n Significant outcomes are more likely to be reported than non-significant outcomes. n Should unpublished data be included in systemic review? n Pre-specified inclusion (quality) criteria are recommended. 12

A Demonstration of Publication Bias Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 13

ph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ Predictors of Publication

Language Bias Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 15

Publication/ Reporting Bias? 16 Pan Z et al. PLo. S Med. 2005 Dec; 2(12): e 334

Meta-Analysis of Observational Studies n In observational studies, bigger is not necessarily better. n This is a danger that meta-analysis of observational data produces precise but spurious results. n A careful systemic review and examination of source of heterogeneity are more important than combining results. n Statistical combination of data should not be the main component of systemic review of observational data. n It is often desirable to have individual level data for this purpose. n Confounding and effect modifiers. 17

Example 1. Fat Intake and Risk of Breast Cancer n Recall bias might play a role in the findings from case-control studies. Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 18

Example 2: Intermittent Sunlight Exposure and Risk of Melanoma n Lack of blinding of the hypothesis to the subjects may introduce recall bias. Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 19

Example 3. Formaldehyde Exposure and Lung Cancer n Dose-response or selection bias? Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 20

Carotenoids and the Risk of Developing Lung Cancer: A Systemic Review*. n Objective: Systemic review of the association between carotenoids and lung cancer. n Included both RCT and prospective observational studies (smoking adjusted). n Examined total carotenoids, β-carotene, α-carotene, β-cyrptoxanthin, lycopene, and lutein-zeaxanthin. n Examined carotenoid supplements, dietary intake of carotenoids, and serum carotenoid concentrations. 21 *Gallicchio L et al, Am J Clin Nutr 2008; 88: 372 -83.

β-carotene Supplement Use in RCT 22 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Dietary Total Carotenoid Intake: Prospective Cohort Studies 23 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Dietary β-carotene Intake: Prospective Cohort Studies 24 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Serum Total Carotenoid Concentrations: Prospective Cohort Studies. 25 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Dose Response 26 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Subgroup Analysis 27 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Method for Systemic Review n n n n 1. Define research questions 2. Define inclusion/exclusion criteria 3. Search the literature 4. Select articles 5. Evaluate the internal/external validity of the studies 6. Extract/Abstract Data 7. Calculate effect size and standard error 8. Examine heterogeneity 9. Assess publication bias 10. Combining study results if appropriate 11. Influence analysis, sensitivity analysis 12. Interpretation of results 13. Reporting 28

Stages of a Systemic Review Stage 1 Planning Stage 2 Conducting Stage 3 Reporting Identification of the need for a systemic review Identification of the research The report and recommendations Development of a proposal Selection of studies Study quality assessment Data extraction Data synthesis 29

1. Clearly Formulated Question n The research question is extremely important! n Get feedback about your specific research question from many people (content expert, savvy clinician in the field, methodologist) n The question should be clearly specified n What is the study objective To validate results in a large population n To guide new studies n Pose questions in both biologic and health care terms specifying with operational definitions n Population n Intervention/exposure n Outcomes (both beneficial and harmful) n 30

Study Protocol n Develop a protocol of what will be done n n Background: why is this important? Specify the research question n Research question defines the following criteria Provide an overview of the methods, including search strategy, inclusion/exclusion criteria, quality assessment, how data will be abstracted, and an analysis plan Specify any subgroup analyses or sensitivity analyses (best if these are a priori) n Initially, inclusion criteria should be overly broad n n E. g. , search all alcohol when specifically interested in wine. E. g. , search all dietary, anthropometric related papers when 31 interested in carotenoids.

2. Develop Inclusion Criteria n Use clinical/scientific judgment to enhance validity and homogeneity n Validity (Study quality) n Homogeneity n Similar study design n Similar patient populations n Similar interventions/exposures n Similar outcomes 32

Typical Inclusion Criteria n Study Design (e. g. , RCT, prospective cohort) n Population (risk group) n Interventions (dosage, regimen)/exposure of interest n Outcomes (definitions, follow-up period) 33

Practical Considerations in Defining Eligibility for a Systemic Review Study designs to be included Years of publication or study conduct Languages Choice among multiple publications Restrictions due to sample size or follow-up duration n Similarity of treatment and/or exposure n Completeness of information n n 34

List of excluded studies Chao, C. et al. Am. J. Epidemiol. 2008 168: 471 -480; doi: 10. 1093/aje/kwn 160 35 Copyright restrictions may apply.

3. Comprehensive Literature Search n Need a well formulated and coordinated effort n Seek guidance from a librarian n Specify language constraints n Requirements for comprehensiveness of the search depends on the field and question to be addressed 36

Searching the Literature n Database searches: MEDLINE, EMBASE, ISI Web of Science, Psych. Lit, Cancer. Lit, Cochrane, Dissertations online n Reference lists of retrieved articles n Manual searching of related journals, conference proceedings, textbooks n Experts in the field n Granting agencies n Study/trial registries n Industry (device manufacturers, pharmaceutical companies) 37

How to Develop a Search Strategy (Example) n Study Question n What is the relationship between consumption of different types of alcoholic beverage (beer, wine and liquor) and risk of lung cancer? n Break down the question into facets (not all may be needed for searching) n n n Exposure: Alcoholic beverage consumption Outcome: Lung cancer Study design: Observational studies with internal comparison group (case-control or cohort) 38

How to Develop a Search Strategy (Cont. ) n Identify synonyms, spelling variants, and subject headings associated with each facet Text terms (title/abstract) Me. SH alcohol ethanol alcoholics beer wine liquor alcohol drinking ethanol 39

How to Develop a Search Strategy (Cont. ) Text terms (title/abstract) Me. SH lung in combination with cancer neoplasm carcinoma tumor incidence mortality lung neoplasm 40

4. Study Selection n Unbiased Study Selection n Two independent reviewers select studies n Based on a priori specification of the population, intervention, outcomes and study design n Differences are resolved by consensus n Specify reasons for rejecting studies n. Keep a record of what is excluded and reason (journals may require this and the search must be reproducible) 41

Figure 1. Flow Diagram of Study Selection Process Potentially relevant references identified after liberal screening of the electronic search (n=#) Excluded by Title/Abstract (n=#) List the reasons Articles retrieved for more detailed evaluation (n=#) Articles excluded after evaluation of full text (n=#) List the reasons Relevant studies included in the metaanalysis (n=#) 42

Carotenoids and Lung Cancer Systemic Review 43 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

Study Quality Assessment Quality of study conduct vs. quality of reporting Controversial since this process is subjective and can introduce bias No standard method to assess quality of a study Scales and checklists exist but can be challenging to incorporate for observational studies n Pre-specify algorithm for quality assessment n Assess quality of each study in uniform, systematic and complete manner n Once quality is assessed then what to do with that information n Consider weighting each study result by quality score n Exclude studies with ‘poor’ quality n Stratifying by methodological quality component (component approach) n n 44

5. Data Extraction n Develop an easy-to-fill-out form for collecting data from each article n n n Study methods (quality) Study description (population) Results (outcomes, side effects) n Two independent reviewers extract data n Should be explicit, unbiased, and reproducible n Include all relevant measures of benefit and harm of the intervention n Contact investigators of the studies for clarification in published methods, data n Differences in data extraction are resolved by consensus 45

Data Extraction: Study Characteristics n Types of publication (journal article, abstract or n n n n unpublished data) Publication year and country of origin Study participants (sample size, age, gender, race, health status) Design details (case-control, cohort, parallel or cross-over, randomization, blinding) Nature of treatment and control Study duration Measurement of compliance Definition and measurement of outcome Other confounders 46

Data Extraction: Study Outcome n Continuous data Outcomes summarized as means (blood pressure, weight) n Dichotomous or binary data n Each individual must be in one of two states (dead/alive, smoking/not smoking) n These data are summarized using odds ratios, risk ratios, or risk differences n Survival or time to event data n Outcome of interest is the time to the occurrence of an event n Usually summarized using hazard ratios n Other data n Some outcomes may be n n n Short ordinal scales (pain scales: individuals’ rate their pain as none, mild, moderate, severe) for which it is not sensible to calculate a mean Event counts (number of asthma attacks per month) 47

48 Gallicchio et al, Am J Clin Nutr 2008; 88: 371 -83

6. Calculating Effect Size and Standard Error n Perform a narrative, qualitative summary when data are too sparse, or too low quality or too heterogeneous to proceed with a meta-analysis n The results from each study are converted into an Odds Ratio (OR) or Effect Size (ES) n 95% confidence intervals (CI) are calculated for each study-specific OR or ES n For a meta-analysis, If only confidence intervals are given, computation will be required to obtain estimates of the standard error 49

7. Examining Heterogeneity n Statistical test for heterogeneity n Visual inspection/Graphical approach n Forest plot, Galbraith plot n Meta-regression n Unit of regression: study n Dependent variable: study-specific effect estimate n Independent variables: study-specific characteristics (e. g. , study design, geographic location, length of follow-up) 50

Examining Forest Plot for Heterogeneity Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 51

Meta-regression STATA Output n Type 2 diabetes and risk of NHL: systemic review and meta- analysis 52

8. Assessing Publication Bias n Results because negative studies are less likely to be submitted or published n Can bias the results of a meta-analysis toward a positive finding n Can evaluate publication bias graphically (funnel plot) or through statistical analysis n Egger’s test, Begg’s test 53

Begg’s Funnel Plot Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 54

Begg's funnel plot with pseudo 95% confidence limits for assessment of publication bias Chao, C. et al. Am. J. Epidemiol. 2008 168: 471 -480; doi: 10. 1093/aje/kwn 160 55 Copyright restrictions may apply.

9. Meta-Analysis for Calculating a Summary Effect Estimate n Several methods are available for combining study results Inverse variance method n M-H methods n Peto’s odds ratio method n n Fixed effect vs. random effect 56

Subgroup Analyses n Pre-specify hypothesis-testing subgroup analyses and keep few in number n Label all posteriori subgroup analyses n When subgroup differences are detected, interpret in light of whether they were: n n n Established a priori Few in number Supported by plausible mechanisms Important (qualitative vs. quantitative) Consistent across studies Statistically significant (adjusted for multiple testing) 57

10. Influence Analysis n Examine how each study influence the summary statistic by removing one study at a time and re-calculate the combined estimate. n A graphically display can be used for visual inspection of influential studies. 58

Influence Analysis STATA Output 59

Influence Analysis STATA Output Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 60

10. Sensitivity Analysis n Test robustness of results relative to key features of the studies and key assumptions and decisions n Include tests of bias due to retrospective nature (e. g. , with/without studies of lower methodological quality) 61

Component Approach Graph adopted from Egger, Smith and Altman. Systemic Reviews in Health Care. BMJ 62

11. Interpretation of Findings n Interpret results in context of clinical practice n State methodological limitations of the individual studies n n included and in the meta-analysis Consider size of effect in studies and meta-analysis, consistency of effect sizes and any dose-response relationship Interpret results in light of other valuable evidence Make recommendations clear and practical Propose future research agenda (clinical and methodological requirements) 63

12. Describe Studies and Findings n Guideline for reporting systemic review n The QUOROM statement (Quality Of Reporting Of Meta-analysis) Moher D, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999, 354(9193): 1896 -900. 64

QUOROM Check List Moher D, et al. Lancet 1999, 354(9193): 1896900. 65

Summary n A well conducted systemic review allows for a more n n objective appraisal of the evidence than traditional narrative reviews Systemic review may contribute to resolve uncertainties and disagreements in original research Meta-analysis may enhance the precision of estimates of treatment effects Exploratory analyses (i. e. , subgroups who are likely to respond well to a treatment) may guide cost effective treatment decisions Systemic review may demonstrate areas where the evidence is inadequate and thus identity areas where further research is needed 66