Systematic Reviews What Why and How Judy Welsh

  • Slides: 88
Download presentation
Systematic Reviews: What, Why, and How Judy Welsh NIH Library welshju@mail. nih. gov August

Systematic Reviews: What, Why, and How Judy Welsh NIH Library welshju@mail. nih. gov August 2017

Learning Objectives 1. What is a systematic review? 2. What is the difference between

Learning Objectives 1. What is a systematic review? 2. What is the difference between a systematic review and a traditional review? 3. Why are systematic reviews important? 4. What are the steps in performing a systematic review?

Learning Objectives 1. What is a systematic review? 2. What is the difference between

Learning Objectives 1. What is a systematic review? 2. What is the difference between a systematic review and a traditional review? 3. Why are systematic reviews important? 4. What are the steps in performing a systematic review?

Cochrane graphic illustration of the systematic review process

Cochrane graphic illustration of the systematic review process

What is a systematic review? • Uses transparent procedures to find, evaluate and synthesize

What is a systematic review? • Uses transparent procedures to find, evaluate and synthesize the results of independent studies. • Procedures are explicitly defined in advance, to assure that the process is transparent and can be replicated. • This process is also designed to minimize bias.

Unique characteristics of a systematic review • A systematic review must have: – –

Unique characteristics of a systematic review • A systematic review must have: – – Clear inclusion and exclusion criteria Explicit search strategy Systematic coding and analysis of included studies Meta‐analysis (where possible)

What is a meta-analysis? • Optional component of a systematic review – A statistical

What is a meta-analysis? • Optional component of a systematic review – A statistical analysis of results from individual studies • Increase power • Improve estimates of the size of the effect • Resolve uncertainty when reports disagree

Traditional vs Systematic Reviews

Traditional vs Systematic Reviews

What is a traditional review? • Uses informal, unsystematic and subjective methods to collect,

What is a traditional review? • Uses informal, unsystematic and subjective methods to collect, interpret, and summarize information. – Searching, quality appraisal and data synthesis are often not documented. – Since scientific methods are not routinely used to identify, assess and synthesize information, assumptions cannot be easily verified. – Greater risk of author and selection bias

Comparison of traditional and systematic reviews Traditional review Systematic review • Subjective • Broad

Comparison of traditional and systematic reviews Traditional review Systematic review • Subjective • Broad question • Methods unclear • • Objective Narrow question Methods clearly defined Studies appraised

Evidence based pyramid Reviews

Evidence based pyramid Reviews

Learning Objectives 1. What is a systematic review? 2. What is the difference between

Learning Objectives 1. What is a systematic review? 2. What is the difference between a systematic review and a traditional review? 3. Why are systematic reviews important? 4. What are the steps in performing a systematic review?

Information overload

Information overload

EBP and clinical practice guidelines • Healthcare decisions should be informed by the best

EBP and clinical practice guidelines • Healthcare decisions should be informed by the best available research evidence. – Many guidelines rely on previously published systematic reviews – Practice guideline quality is dependent on rigorous systematic review methods and high quality primary studies • • Primary studies Randomized controlled trials Systematic reviews Evidence based practice and clinical practice guidelines

Known and unknown areas of study • SRs can show which treatments and prevention

Known and unknown areas of study • SRs can show which treatments and prevention methods have been proven to work - and what remains unknown. • SRs are important for pointing to areas where more research is needed. • Systematic reviews are the basis for what is often called evidence-based medicine or health care.

Learning Objectives 1. What is a systematic review? 2. What is the difference between

Learning Objectives 1. What is a systematic review? 2. What is the difference between a systematic review and a traditional review? 3. Why are systematic reviews important? 4. What are the steps in performing a systematic review?

Stages of a systematic review Develop a focused research question Define inclusion and exclusion

Stages of a systematic review Develop a focused research question Define inclusion and exclusion criteria Search the literature Select studies Assess study quality Extract data Analyze and present results Interpret results and draw conclusions Update as necessary

Stages of a systematic review Develop a focused research question Define inclusion and exclusion

Stages of a systematic review Develop a focused research question Define inclusion and exclusion criteria Search the literature Select studies Assess study quality Extract data Analyze and present results Interpret results and draw conclusions Update as needed

Research Question • A clearly defined, focused systematic review begins with a well formulated

Research Question • A clearly defined, focused systematic review begins with a well formulated research question.

Research Question • The research question guides the author in working through many stages

Research Question • The research question guides the author in working through many stages of the systematic review process – – – Defining inclusion and exclusion criteria Searching the literature Selecting studies Extracting data Analyzing and presenting results

FINER Criteria • A research question should be: – – – Feasible Interesting Novel

FINER Criteria • A research question should be: – – – Feasible Interesting Novel Ethical Relevant

PICO Framework • A well-established format for structuring research questions is known by the

PICO Framework • A well-established format for structuring research questions is known by the acronym PICO. – – Patient or Population Intervention or Indicator Comparator or Control Outcome

PICO • • Patients Intervention Comparison Outcome • • Patients undergoing upper abdominal surgery

PICO • • Patients Intervention Comparison Outcome • • Patients undergoing upper abdominal surgery Prophylactic physical therapy No prophylactic physical therapy Prevent post operative pulmonary complications

Types of clinical research questions • Many types of research questions can be expressed

Types of clinical research questions • Many types of research questions can be expressed using PICO components. – – – Therapy Diagnosis Prognosis Etiology / Harm Clinical Prediction Guides

Stages of a systematic review Develop A focused research question Define inclusion and exclusion

Stages of a systematic review Develop A focused research question Define inclusion and exclusion criteria Search the literature Select studies Assess study quality Extract data Analyze and present results Interpret results and draw conclusions Update as needed

Inclusion and exclusion criteria • One of the features that distinguish a systematic review

Inclusion and exclusion criteria • One of the features that distinguish a systematic review from a traditional review is the pre‐ specification of inclusion and exclusion criteria.

Inclusion criteria • Inclusion criteria are a combination of • Aspects of the research

Inclusion criteria • Inclusion criteria are a combination of • Aspects of the research question • • Population Intervention Comparison Outcome • Study type • Randomized controlled trials • Observational studies

Stages of a systematic review Develop A focused research question Define inclusion and exclusion

Stages of a systematic review Develop A focused research question Define inclusion and exclusion criteria Search the literature Select studies Assess study quality Extract data Analyze and present results Interpret results and draw conclusions Update as needed

Search the literature • The goal of the literature search is to discover all

Search the literature • The goal of the literature search is to discover all studies that meet the inclusion criteria – Search comprehensively • Terminology • Databases – Search for grey literature • Not commercially published – Search for unpublished studies • Reduce risk of publication bias

Recommended databases • The Cochrane Collaboration recommends searching the following databases (at minimum): –

Recommended databases • The Cochrane Collaboration recommends searching the following databases (at minimum): – Pub. Med – EMBASE – Cochrane Central Register of Controlled Trials

Supplementary databases • Interdisciplinary databases – Scopus – Web of Science • Specialized databases

Supplementary databases • Interdisciplinary databases – Scopus – Web of Science • Specialized databases – CINAHL Plus – PEDro: Physiotherapy Evidence Database – Psyc. INFO

Search strategy • The search strategy should be designed to identify the maximum number

Search strategy • The search strategy should be designed to identify the maximum number of studies relevant to the research question. – The search strategy should be systematic, transparent and reproducible. – Database specific controlled vocabulary terms and all relevant text words should be included in the search strategy.

Supplementary searching techniques • Search cited and citing references – Scopus – Web of

Supplementary searching techniques • Search cited and citing references – Scopus – Web of Science

Supplementary searching techniques • Hand search selected journals and conference proceedings • Conduct author

Supplementary searching techniques • Hand search selected journals and conference proceedings • Conduct author searches for recent articles written by topic experts

Unpublished and grey literature • The inclusion of unpublished and grey literature may minimize

Unpublished and grey literature • The inclusion of unpublished and grey literature may minimize the potential effects of publication bias. – Publication bias • Occurs when the outcome of an experiment or research study influences the decision about whether—or how quickly—the manuscript may be published

What is grey literature? • Grey literature refers to academic, business, government or industry

What is grey literature? • Grey literature refers to academic, business, government or industry print or electronic literature that is not controlled by commercial publishers. – – – Conference proceedings Research reports Government reports Dissertations, theses Research monographs Organization websites

Sources of grey literature • Conference proceedings – – EMBASE Scopus Web of Science

Sources of grey literature • Conference proceedings – – EMBASE Scopus Web of Science Google

Sources of unpublished literature • Clinical trials – – – Clinical. Trials. gov Centerwatch.

Sources of unpublished literature • Clinical trials – – – Clinical. Trials. gov Centerwatch. com EU Clinical Trials Register ISRCTN Registry Open. Trials WHO International Clinical Trials Registry Platform

Compile search results • Compile search results using reference management software (End. Note or

Compile search results • Compile search results using reference management software (End. Note or Mendeley) • Remove duplicate records

Document search process • Document the search process – – Databases Dates searched Search

Document search process • Document the search process – – Databases Dates searched Search strategies Limits (date ranges, publication types, language restrictions)

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search The Literature Select studies Assess Study Quality Extract Data Analyze And Present Results Interpret Results And Draw Conclusions Update As Needed

Select studies • Identification of studies meeting inclusion criteria should be done independently by

Select studies • Identification of studies meeting inclusion criteria should be done independently by two review authors. – Review titles and abstracts of retrieved citations. – Review full text of studies which are found to meet the inclusion criteria. – Keep a record of reasons for inclusion or exclusion.

Tools for sorting • Tools for sorting – Reference management software • End. Note

Tools for sorting • Tools for sorting – Reference management software • End. Note – New web application • Rayyan – Excel

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search the Literature Select Studies Assess study quality Extract Data Analyze And Present Results Interpret Results And Draw Conclusions Update As Needed

Assess study quality • Following the full text review, assess the selected studies for

Assess study quality • Following the full text review, assess the selected studies for risk of bias and study quality.

Tools for assessing risk of bias • Tools for assessing risk of bias –

Tools for assessing risk of bias • Tools for assessing risk of bias – ROBIS: Risk of Bias in Systematic Reviews – ROBINS‐I tool: Risk of Bias in Non‐randomized studies of Interventions – Ro. B 2. 0 tool (revised tool for Risk of Bias in randomized trials

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search the Literature Select Studies Assess Study Quality Extract data Analyze And Present Results Interpret Results And Draw Conclusions Update As Needed

Extract data • Extract reported findings from selected studies using a data extraction form.

Extract data • Extract reported findings from selected studies using a data extraction form. – Extraction forms and approaches should be determined by the needs of the specific review. – At least two review authors should independently extract data from study reports.

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search the Literature Select Studies Assess Study Quality Extract Data Analyze and present results Interpret Results And Draw Conclusions Update As Needed

Analyze and present results • The findings from individual studies are aggregated to produce

Analyze and present results • The findings from individual studies are aggregated to produce a type of evidence synthesis appropriate to the type of data within the review. – Narrative synthesis – findings are summarized and explained in words – Quantitative/statistical synthesis – data from individual studies are combined statistically and then summarized (meta‐analysis)

Tables and Figures • Tables and figures are used to present included studies and

Tables and Figures • Tables and figures are used to present included studies and their findings in a systematic and clear format. – Flow diagram – Summary of findings table – Forest plot

PRISMA Flow diagram • PRISMA Flow diagram – Depicts the flow of information through

PRISMA Flow diagram • PRISMA Flow diagram – Depicts the flow of information through the different phases of a systematic review – Documents the number of studies that remain after each stage of the selection process – Maps the number of studies identified, included and excluded, and the reasons for being excluded – PRISMA Flow Diagram Generator

PRISMA Flow Diagram

PRISMA Flow Diagram

Summary of findings table • Summary of findings table – Provides key information concerning

Summary of findings table • Summary of findings table – Provides key information concerning the quality of evidence – Depicts the magnitude of effect of the interventions – Illustrates the sum of available data on all important outcomes for a given comparison Cochrane Handbook

Summary of findings table using GRADE methodology

Summary of findings table using GRADE methodology

Forest Plot • Forest Plot – A graphical display designed to illustrate the relative

Forest Plot • Forest Plot – A graphical display designed to illustrate the relative strength of treatment effects in multiple quantitative scientific studies addressing the same question (meta‐ analysis)

Forest Plot Names of fictional studies on left; odds ratios and confidence intervals on

Forest Plot Names of fictional studies on left; odds ratios and confidence intervals on right; odds ratios (squares proportional to weights used in meta-analysis); summary measure (center line of diamond); associated confidence intervals (lateral tips of diamond); solid vertical line of no effect

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search the Literature Select Studies Assess Study Quality Extract Data Analyze And Present Results Interpret results and draw conclusions Update As Needed

Interpret results and draw conclusions • Statement of findings, discussion and conclusions – Information

Interpret results and draw conclusions • Statement of findings, discussion and conclusions – Information on all important outcomes, including adverse outcomes – Quality of evidence for each outcome – How values and preferences may bear on balance of benefits, harms, and costs of interventions

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion

Stages of a systematic review Develop A Focused Research Question Define Inclusion And Exclusion Criteria Search the Literature Select Studies Assess Study Quality Extract Data Analyze And Present Results Interpret Results And Draw Conclusions Update as necessary

Whether and when to update • Decisions about whether and when to update a

Whether and when to update • Decisions about whether and when to update a systematic review – – The currency of the question asked The need for updating to maintain credibility The availability of new evidence Whether new research or new methods will affect the findings When and how to update systematic reviews, BMJ (2016)

Cochrane graphic illustration of the systematic review process

Cochrane graphic illustration of the systematic review process

Registries, tools and archives • Cochrane Collaboration – Cochrane Library • Cochrane protocols, systematic

Registries, tools and archives • Cochrane Collaboration – Cochrane Library • Cochrane protocols, systematic reviews, other reviews and trials – Guides and handbooks • • Cochrane Handbook for Systematic Reviews of Interventions Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Cochrane Standards for conduct and reporting of new reviews of interventions GRADE Handbook – Cochrane tools for assessing risk of bias • ROBIS: Risk of Bias in Systematic Reviews • ROBINS‐I tool: Risk of Bias in Non‐randomized studies of Interventions • Ro. B 2. 0 tool (revised tool for Risk of Bias in randomized trials – Methodological Expectations of Cochrane Intervention Reviews

Registries, tools and archives • GRADE Working Group – GRADE (Grading of Recommendations, Assessment,

Registries, tools and archives • GRADE Working Group – GRADE (Grading of Recommendations, Assessment, Development and Evaluation) • Method of grading quality of evidence and strength of recommendations in guidelines • GRADEpro software can be used to create Summary of Findings tables

Registries, tools and archives • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)

Registries, tools and archives • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) – PRISMA checklist • 27 items for inclusion in systemic reviews and meta-analyses – PRISMA flow diagram • Diagram depicting flow of information and number of records through phases of a systematic review • Prospero – International prospective register of health‐related systematic reviews

Software for systematic reviews • Software – Software for Systematic Reviewing – Software Programs

Software for systematic reviews • Software – Software for Systematic Reviewing – Software Programs for Preparing and Maintaining Systematic Reviews – Systematic Review Toolbox

Special Services offered at the NIH Library: A quick overview Judith Welsh, BSN, MLS

Special Services offered at the NIH Library: A quick overview Judith Welsh, BSN, MLS NIH Library August 2017

Learning Objectives 1. What special services are offered at the NIH Library?

Learning Objectives 1. What special services are offered at the NIH Library?

Special Services 3 D Printing—Verma Walker Bibliometrics—Chris Belter; Ya‐Ling Lu Bioinformatics—Lynn Young Data Services—Lisa

Special Services 3 D Printing—Verma Walker Bibliometrics—Chris Belter; Ya‐Ling Lu Bioinformatics—Lynn Young Data Services—Lisa Federer Editing—Cindy Clark – Plagiarism Checking software • End. Note site license • Technology Hub—Doug Joubert • Translations—Monica Valencia; Doug Doty • • •

3 D Printing and Modeling • NIH Library offers free 3 D printing to

3 D Printing and Modeling • NIH Library offers free 3 D printing to NIH staff • Printers – – – Models must be used for NIH research Self service First come, first served 60 minute orientation required Located in Technology Hub https: //nihlibrary. nih. gov/services/3 d‐printing‐service

3 D Printing and Modeling • Modeling software – Several open source and commercial

3 D Printing and Modeling • Modeling software – Several open source and commercial 3 D modeling software packages are available on Technology Hub computers • Reservations are required • NIH 3 D Print Exchange – Virtual collection of bioscientific 3 D models and tutorials for 3 D printing • http: //3 Dprint. nih. gov • Prosthetics Collection Ribbon Structure

Bibliometrics Service Program • Provides publication analysis services to NIH staff – – –

Bibliometrics Service Program • Provides publication analysis services to NIH staff – – – Measure productivity Detect collaboration Identify research topics Assess citation impact https: //nihlibrary. nih. gov/services/bibliometrics

Bibliometrics Service Program • Measure productivity – Identify publications by scientists, laboratories, and grant

Bibliometrics Service Program • Measure productivity – Identify publications by scientists, laboratories, and grant programs – How much research? Change over time?

Bibliometrics Service Program • Detect collaboration – Generate individual and institutional co‐authorship networks –

Bibliometrics Service Program • Detect collaboration – Generate individual and institutional co‐authorship networks – Which scientists and institutions are working together?

Bibliometrics Service Program • Identify research topics – Analyze topical and structural links between

Bibliometrics Service Program • Identify research topics – Analyze topical and structural links between papers – How are papers by a lab, program, or discipline related? Change over time?

Bibliometrics Service Program • Assess citation impact – Calculate citation scores – How useful

Bibliometrics Service Program • Assess citation impact – Calculate citation scores – How useful has research published by your lab, division, or program been to other scientists?

Topic analysis for RMD : Chris Belter Dots in the map represent papers by

Topic analysis for RMD : Chris Belter Dots in the map represent papers by the Rehab Medicine Department, dot size indicates the paper’s times cited count, and dot color indicates the paper’s topic as described on the right of the topic page. Lines indicate shared references between the connected papers.

Bioinformatics Support Program • Supports NIH researchers by assisting in the analysis and interpretation

Bioinformatics Support Program • Supports NIH researchers by assisting in the analysis and interpretation of genomic data to reveal the molecular mechanisms of disease Consultations Training Tutorials Powerful tools and bioinformatics resources are available via the NIH Library website and Bioinformatics workstations located in the library reading room – https: //nihlibrary. nih. gov/services/bioinformatics‐support – –

Data Services Program • The Data Services program provides assistance to NIH investigators throughout

Data Services Program • The Data Services program provides assistance to NIH investigators throughout the research process from project planning to post project preservation – Locating existing data and processing datasets for reuse – Organizing and describing data to facilitate analysis, collaboration, and sharing – Data mining and visualization – Retention, long term preservation, and sharing data – https: //nihlibrary. nih. gov/services/data‐services

Data Services Training and Classes • Training and classes – Classes, tutorials and consultations

Data Services Training and Classes • Training and classes – Classes, tutorials and consultations are offered on a variety of data‐related topics from basic data management to more in depth classes on specialized topics. • General data management courses • R statistical programming courses • R data visualization programming courses

Data Sciences Workstation • Data Sciences Workstation – The Technology Hub includes a Data

Data Sciences Workstation • Data Sciences Workstation – The Technology Hub includes a Data Sciences Workstation and collaboration pods • Tools for data analysis, processing and visualization of different types of data

Data Visualization Service • The Data Visualization Service provides – Access to licensed and

Data Visualization Service • The Data Visualization Service provides – Access to licensed and open source software – Use of a large touch display for visualizing data • 65" interactive digital display used to visualize and demonstration data – Data visualization community of practice

End. Note X 8 • The NIH Library has acquired a one-year enterprise wide

End. Note X 8 • The NIH Library has acquired a one-year enterprise wide license for End. Note through December 31, 2017. • This license allows anyone directly associated with NIH to download and/or upgrade to the latest edition of End. Note. • The software can be installed up to three computers per user—work and home. – https: //nihlibrary. nih. gov/about-us/news/endnote-x 8 -now -available

Technology Hub • The Technology Hub – Experience ways to use cutting edge technology

Technology Hub • The Technology Hub – Experience ways to use cutting edge technology • • 3 D Printing and Modeling Data Visualization Touchscreen Collaborative Workspaces Digital Production Studio Smartpens Virtual Reality https: //nihlibrary. nih. gov/services/technology-hub

Translations • NIH Library Translations Office – Work related documents • Personal documents •

Translations • NIH Library Translations Office – Work related documents • Personal documents • Medical and Scientific documents • Articles • Service Costs – Free • • • From English to Russian From German to English From Spanish to English From Italian to English From French to English – All other translation rates • 15 to 25 cents per word

Writing Center and Editing Service • NIH Library Writing Center – Virtual access •

Writing Center and Editing Service • NIH Library Writing Center – Virtual access • • • – – – Grammar and punctuation Journal impact factor H-index Copyright Publishing process Physical Writing Center Laptops Editing service Plagiarism checking service https: //nihlibrary. nih. gov/services/editing