RAPID SYSTEMATIC REVIEWS Alex Salam Entebbe 9 th
RAPID SYSTEMATIC REVIEWS Alex Salam Entebbe 9 th March 2017
OUTLINE • Key elements in a rapid systematic review • How rapid reviews differ from standard SR • Formulating a relevant research question • Searching the literature • Extracting data • Appraising quality
WHAT IS A SYSTEMATIC REVIEW? • A literature review that asks a specific question and has: • A clearly stated set of objectives • Pre-defined eligibility criteria for studies • An explicit, reproducible search methodology • A systematic aim to identify all studies • An assessment of the validity of the findings of the studies • A systematic presentation, and synthesis, of the characteristics and findings of the studies
STEPS IN A SYSTEMATIC REVIEW 1. Formulate research question 2. Determine inclusion and exclusion criteria 3. Literature search 4. Pilot test inclusion criteria 5. Screening – Title/Abstract 6. Screening – Full-text 7. Develop and test data abstraction form 8. Abstract data 9. Study appraisal 10. Analysis (meta-analysis) 11. Synthesis 12. Prepare Manuscript 13. Disseminate Findings
STEPS IN A SYSTEMATIC REVIEW? http: //utas. libguides. com/Systematic. Reviews
WHY DO A SYSTEMATIC REVIEW? • Not doing so: • Risks doing research for which the answer is already known • Limits opportunities to learn from previous successes and failures when designing your study • Might miss opportunities to design your study • Doing so: • Will help you design your study • Make it easier to discuss your eventual findings
WHY NOT JUST A NARRATIVE REVIEW? FEATURE NARRATIVE SYSTEMATIC Question Often broad in scope Often focused clinical Q Search method Not usually specified Potentially biased Predefined protocol-based Explicit search strategy Inclusion of studies Not usually specified Potentially biased Predefined selection criteria Uniformly applied Search media Mainly Pub. Med or Medline Diverse Data extraction Simple descriptions Protocol based Statistical values Data quality Descriptive Grading by guidelines Synthesis Descriptive Based on guidelines such e. g. PRISMA Inferences Easily biased Based on data included Cook DJ et al. Ann intern med. 1997 Mar 1; 126(5): 376 -80.
WHY NOT JUST A NARRATIVE REVIEW? AUTHOR SEARCH METHOD Kane et al, None 1999 Narrative RCT 2 Duggan et - Biological abstracts 20 - Cochrane Library al, 1999 Systematic - Cochrane Schizoph. Group’s Register - Embase - Medline - Psyc. LIT - By contacting companies & authors Cipriani A et al. Epidemiol Psichiatr Soc. 2003 Sep 1; 12: 146 -53. CONCLUSIONS “The atypical drug, olanzapine, with a favourable toxicity profile and proven II III efficacy, offers a promising alternative to previous treatments” “The very great losses to follow up make recommendations difficult. Global impression suggests that 10 -15 mg/day of olanzapine is antipsychotic, being better than placebo, but, for those with severe illness, when compared to typicals and atypicals there is little difference for the same outcome. On one sub-scale score, of one mental state rating scale, olanzapine shows superiority over typical antipsychotics for negative symptoms. This result is difficult to interpret clinically. Such findings need replication in large simple studies and should not form the evidence base of treatment recommendations. Olanzapine may have fewer extrapyramidal effects than chlorpromazine and haloperidol and, perhaps, than risperidone. ”
PROBLEMS WITH A SYSTEMATIC REVIEW • A SR take 6 months to 2 years and usually require a budget of at least $100, 000. Month 1 -2 3 -8 2 -3 3 -8 3 3 -10 5 -11 8 -10 1 -11 12 - Preparation of protocol Searches for published and unpublished studies Pilot test of eligibility criteria Inclusion assessments Pilot test of ‘Risk of bias’ assessment Validity assessments Data collection Pilot test of data collection Data entry Follow up of missing information Analysis Preparation of review report Keeping the review up-to-date Petticrew M et al. John Wiley & Sons; 2008 Apr 15.
PROBLEMS WITH A SYSTEMATIC REVIEW Figure 3. Publication dates of SARS papers in the 11 research domains (311 articles) Xing W et al. PLOS Med. 2010.
SOLUTION? • Rapid systematic review – Literature review produced using accelerated and streamlined systematic review methods: 1. Narrow search strategy • Eliminate consultation with experts • Limit number of databases searched (by study design, language, dates) • Limit or omit grey literature • Limit or omit hand searching of reference lists and relevant journals • Narrow time frame for article retrieval • Use non-iterative search strategy 2. Limit number of reviewers involved in: • Inclusion/exclusion • Data extraction • Quality assessment
DIFFERENCE RAPID REVIEW SYSTEMATIC REVIEW TIMEFRAME < 5 weeks 6 months to 2 years QUESTION Question specific a priori (may include broad PICOS) Often focused clinical question SOURCES & SEARCHES Sources may be limited but search strategy is explicit Comprehensive sources searched & explicit strategies SELECTION Criterion-based; uniformly applied Criterion-based APPRAISAL Rigorous, critical appraisal (SR only) Rigorous; critical appraisal SYNTHESIS Descriptive Qualitative summary +/- summary/categorisation of data meta-analysis INFERENCES Limited/cautious interpretations Evidence-based of data
RAPID REVIEW v SYSTEMATIC REVIEW Tricco AC
APPROACHES TO RAPID REVIEW Search limit Screening Data abstraction Risk of bias appraisal 1 >1 database, published only Both date and language One reviewer One person abstracts, other verifies One person assesses, other verifies 2 Updating the literature search of a previous review, published only None One reviewer Not performed 3 >1 database, grey Both date literature and language One reviewer Not performed 4 >1 database, grey Either date literature or language One reviewer Not performed 5 >1 database, grey literature One reviewer Approach Literature search Date Tricco AC et al. J clin epidemiol. 2016 Feb 29; 70: 61 -7.
APPROACHES TO RAPID REVIEW Approach 1 2 3 4 5 Feasibility Timeliness Comprehensiveness Risk of Bias 1 st 2 nd 4 th 1 st 2 nd 1 st 5 th 3 rd 2 nd 4 th 5 th 1 st 3 rd Delphi ranking of different approaches. Ranked based on the distribution of "very" and "extremely" on the 7 -point Likert scale, except. Risk of Bias was ranked on distribution of “not at all” and “very”. Tricco AC et al. J clin epidemiol. 2016 Feb 29; 70: 61 -7.
DOING A RAPID REVIEW http: //utas. libguides. com/Systematic. Reviews
WHAT’S THE POINT OF A RESEARCH QUESTION? • Defines the nature and scope of the review • Identifies the keywords (together with the scoping search) • Determines the search strategy and the search to be undertaken • Provides guidance for selecting the primary research papers needed • Guides the data extraction and synthesis of the results.
TYPES OF RESEARCH QUESTIONS Xing W et al. PLOS Medicine 2010 7(5): e 1000272.
DEVELOPING YOUR RESEARCH QUESTION • ‘Free-form’ question • Defined by simple language • Sometimes very vague • Describes the query that you are interested in e. g. what effect does statin use have on pregnancy?
DEVELOPING YOUR RESEARCH QUESTION • Creating a structured research question: PICO(S) • Populations • Interventions (or exposure) • Comparators • Outcomes • Study design
DEVELOPING YOUR RESEARCH QUESTION • POPULATIONS - The patient population or disease of interest • How is disease/condition defined? • What are the most important characteristics that describe the population? • Any relevant demographic factors (e. g. age, sex, ethnicity)? • What is the setting (e. g. hospital, community etc)? • Are there other types of patient who should be excluded (because they are likely to react to the intervention in a different way)? Example: pediatric patient
DEVELOPING YOUR RESEARCH QUESTION • INTERVENTIONS/EXPOSURES - The intervention or range of interventions of interest • What is the experimental intervention of interest? • Does the intervention have variations? • Are all variations to be included? • How will trials including only part of the intervention be handled? • How will trials including the intervention of interest combined with another intervention be handled? Effect of dexamethasone
DEVELOPING YOUR RESEARCH QUESTION • COMPARATORS - What the intervention should be compared against. • What is the comparison of interest? • Does the comparison have variations? • Are all variations to be included? • How will trials including only part of the comparison be handled? • How will trials including the comparison of interest combined with another intervention be handled? Example: effect of ondansetron
DEVELOPING YOUR RESEARCH QUESTION • OUTCOMES - Outcome of interest. • Morbidity, mortality, survival, health resource use, quality of life, behaviour • Also actual adverse effects. • Should be essential for decision-making and have an emphasis on patient-important outcomes. • Primary outcomes • Secondary outcomes • Consider outcomes relevant to all potential decision makers, including economic data. Example: preventing nausea
DEVELOPING YOUR RESEARCH QUESTION • TIME - Timeframe of outcome measure. • Timeframe is optional in the PICO(T) framework. • Specifying a timeframe may not always be applicable to the study question Example: one week
DEVELOPING YOUR RESEARCH QUESTION • STUDY DESIGNS • ‘major role in determining the reliability of the results’ (CRD) • RCTs usually the study design of choice for effectiveness reviews • Scoping search will help you decide whether to limit by study type • Depends entirely on the nature of your topic
DEVELOPING YOUR RESEARCH QUESTION www. mheducation. co. uk
DEVELOPING YOUR RESEARCH QUESTION • What effect does statin use have on pregnancy? • In pregnant women, does daily statin use compared to no statin increase the risk of congenital malformations in children (RCTs)? Population: pregnant women Intervention: statins Comparator: none Outcome: congenital malformations in the child Study design: NA
DEVELOPING YOUR RESEARCH QUESTION • What is the best way to treat recurrent furuncolosis? • In patients with recurrent furunculosis, do prophylactic antibiotics, compared to placebo, reduce the recurrence rate? Population: patients with recurrent furuncolosis Intervention: prophylactic antibiotics Comparator: placebo Outcome: Reduction in recurrence rate of furuncolosis Study design: RCTs
DEVELOPING YOUR RESEARCH QUESTION www. mheducation. co. uk
DEVELOPING YOUR RESEARCH QUESTION Choice of participants e. g. corticosteroid injection for shoulder tendonitis (narrow) or corticosteroid injection for any tendonitis (broad) Broad scope Narrow scope Advantages: Comprehensive summary of the evidence. Ability to assess generalizability of findings across types of participants Advantages: Manageability for review team. Ease of reading. Disadvantages: May be more appropriate to prepare an overview of reviews. Searching, data collection, analysis and writing may require more resources. Risk of ‘mixing apples and oranges’ (heterogeneity) Interpretation may be difficult. Disadvantages: Evidence may be sparse. Findings may not be generalizable to other settings or populations. Scope could be chosen by review authors to produce a desired result. http: //utas. libguides. com/Systematic. Reviews
DEVELOPING YOUR RESEARCH QUESTION Definition of an intervention e. g. supervised running for depression (narrow) or any exercise for depression (broad) Broad scope Narrow scope Advantages: Comprehensive summary of the evidence. Ability to assess generalizability of findings across different implementations of the intervention. Advantages: Manageability for review team. Ease of reading. Disadvantages: Searching, data collection, analysis and writing may require more resources. Risk of ‘mixing apples and oranges’ (heterogeneity); interpretation may be difficult. Disadvantages: Evidence may be sparse. Findings may not be generalizable to other formulations of the intervention. Scope could be chosen by review authors to produce a desired result. http: //utas. libguides. com/Systematic. Reviews
DEVELOPING YOUR RESEARCH QUESTION Choice of interventions and comparisons e. g. alarms for preventing bedwetting (narrow) or interventions for preventing bedwetting (broad) Broad scope Narrow scope Advantages: Comprehensive summary of the evidence. Advantages: Manageability for review team. Clarity of objectives and ease of reading. Disadvantages: May have limited value when not included in an Overview. Disadvantages: May be unwieldy, and more appropriate to present as an Overview of reviews. Searching, data collection, analysis and writing may require more resources. http: //utas. libguides. com/Systematic. Reviews
DEVELOPING YOUR RESEARCH QUESTION Hypothetical examples of potential for bias based on inadequately defined PICOs Agency for Healthcare Research and Quality. 2008.
SETTING ELIGIBILITY CRITERIA http: //utas. libguides. com/Systematic. Reviews
SETTING ELIGIBILITY CRITERIA • Use your research question and the PICO structure to set the criteria • Consider types of study you want in your review • RCTs, observational (cohort, case controls, case series, registries/databases) • Restrictions on n • Eligibility criteria don’t always have to match your question perfectly
PERFORMING THE SEARCH http: //utas. libguides. com/Systematic. Reviews
PERFORMING THE SEARCH - SOURCES • One of the areas where you can save time in a rapid review • Involve a librarian or information specialist if you can • The types of study you are looking for will to a large extent determine the sources you will search. • Bibliographic databases § Pub. Med/MEDLINE (health) § EMBASE (health) § CINAHL (nursing and allied health) § LILACS (South American literature) § Li. SSa (French literature) § Psych. INFO (psychology)
PERFORMING THE SEARCH - SOURCES • Research and trial registers § Cochrane Central Register of Controlled Trials (CENTRAL) § US National Institutes of Health register (clinicaltrials. gov) § WHO International Clinical Trial Registries Platform LILACS § NHS Research Register
PERFORMING THE SEARCH - SOURCES • Conference proceedings § Mednar § Psyc. EXTRA § Oalster § SIGLE § Google Scholar
PERFORMING THE SEARCH - SOURCES • Theses and dissertations § Index to Thesis § Networked Digital Library of Theses and Dissertations (NDLTD) § Pro. Quest Dissertations and Theses Database § Electronic Thesis Online System (ETh. OS) § TROVE (ADTP)
PERFORMING THE SEARCH - SOURCES • Government websites § NICE (UK) § CDC (US) § MRC (UK) § AHRQ (US) § AIHW (Australia)
PERFORMING THE SEARCH – SEARCH TERMS • Librarian or information specialist very helpful • Sensitive search strategy – ability to identify all relevant studies • Specific search strategy – ability to exclude irrelevant studies • High sensitivity will generally lead to low specificity Consider: • Synonyms: are there other words with similar meanings? • Spelling: can your search term be spelt in different ways? • Singulars and plurals: usually people and things are plural, ideas are expressed as singular. • Truncating terms: using a symbol such as * $ or ? For example, politi* can search for politic, politics and political. • Combining terms using AND, OR and NOT.
PERFORMING THE SEARCH – SEARCH TERMS AND Diabetes AND hypertension OR Diabetes OR hypertension NOT Diabetes NOT hypertension
PERFORMING THE SEARCH – SEARCH TERMS http: //utas. libguides. com/Systematic. Reviews
PERFORMING THE SEARCH – SEARCH TERMS & LIMITS http: //utas. libguides. com/Systematic. Reviews
PERFORMING THE SEARCH - TIPS • Tips • Using the search strategies, or parts of the search strategies from existing systematic reviews. • Keep notes of searches and results (software exists) • Scope and Re-Scope • Test and re-test- gold standard • Avoid ‘scope creep’ (Booth 2011) • Accept the ‘point of diminishing returns’ • Know when to STOP • Searching is never fully transparent, nor reproducible but make it rigorous.
PERFORMING THE SEARCH – RAPID REVIEW • Limiting your search for a rapid review • Focusing on a particular time frame • Limiting the languages that you search • Limiting by study design • Limit the sources
PERFORMING THE SEARCH – SCREENING
PERFORMING THE SEARCH – SCREENING • Titles and abstracts • Estimate the time • Start with a pilot phase • Full text articles • Time consuming • Consider using 1 person for rapid review • Consider how much detail you record regarding rejection • You may not be able to access some articles
PERFORMING THE SEARCH – EXTRACTING
EXTRACTING DATA • Summarise studies in a standardised format to: • Facilitate synthesis and coherent presentation of data • To identify numerical data for meta-analyses • To obtain information and data to objectively assess risk of bias • To identify systematically missing or incorrectly assessed data, outcomes that are not studied and underrepresented populations. • Various electronic tools exist. • Use PICO framework to help choose data elements.
EXTRACTING DATA • STUDY CHARACTERISTICS • Study design (RCT, observational cohort etc) • Participant characteristics (demographics, sample size) • Method of randomisation and blinding • Outcomes (and interventions, if applicable) • Information needed for quality assessment (comparability of groups, exclusions made, length of follow-up, etc) • Your comments on methodology, limitations, generalisability that you have after reading the paper
EXTRACTING DATA • OUTCOME MEASURES • Effect sizes (e. g. Cohen’s d, odds ratio, hazard ratio) • Mean, SD, median, CI, n in each group • Whatever is available and relevant e. g. other statistical tests, graphs, author’s descriptions)
ASSESSING QUALITY • Risk of bias
ASSESSING QUALITY • Risk of bias Cochrane handbook
ASSESSING QUALITY • Generalisability • Statistical issues • How participants were identified and recruited (e. g. accuracy of diagnostic tests) • How the interventions or exposures were defined • How outcomes were measured • Validity of outcome instruments • Length of follow up TOOLS EXIST (Cochrane Risk of Bias tool, ROBINS-I tool)
WRAPPING IT UP • Reporting the review • Prioritising topics for future research • Involving stakeholders
USING AN EXISTING SR • Search • Assess quality (e. g. AMSTAR) • Assess whether up to date
SUMMARY • Check if SR exists • If not RR • Framing the research question in important • Various strategies exist for RR
QUESTIONS?
- Slides: 61