Does it Matter Timeliness or Accuracy of Results

Does it Matter — Timeliness or Accuracy of Results? Results of a Research Program on Rapid Reviews Andrea C. Tricco Ph. D MSc CADTH Symposium 2015

Conflict of interest § I have no actual or potential conflict of interest in relation to this presentation. 2

Outline § § Background Scoping Review of Rapid Reviews International Survey of Rapid Review Producers International Consensus-building Exercise with Key Stakeholders § Ultimate Goal of Our Research Program § Questions 2

Background § Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [Khangura 2012] § Evidence suggests that decision-makers are currently using rapid reviews to inform their decision-making processes § Few studies have examined the methodological characteristics of rapid reviews § We aimed to conduct a research program on rapid reviews to clarify the methods and perceptions of rapid review approaches 3

Methods project 1: A scoping review of rapid review methods Submitted to BMC Medicine

Objective & methods § Objective – To examine rapid review approaches, guidance, impact, and comparisons through a scoping review § Methods – Used methodologically rigorous scoping review methods proposed by Arksey and O’Malley (2005) – MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review producers, and reference lists were searched to identify articles for inclusion – Two reviewers independently screened citations and full-text articles, and abstracted data 5

Results N=3397 citations from MEDLINE, EMBASE, Cochrane Library, previous systematic reviews, and grey literature N=3135 excluded titles and abstracts N=262 potentially relevant full-text articles N=161 excluded full-text reports N=100 rapid reviews (plus 1 companion report) 6 Study flow figure

Results (continued) 7 Word cloud figure for the frequency of terms

Results (continued) Study Characteristics 1997 -2000 2001 -2004 Year of Publication 2005 -2008 2009 -2012 2013 Not Reported Europe (including UK) North America (Canada & United States) Corresponding author’s continent 8 No. of Rapid Reviews (n=100) 2 10 30 51 5 4 58 20 Australia 15 Multiple continents Asia South America Not Reported 3 1 1 2 Study characteristics

Results (continued) Study Characteristics Article Type Topic of Review Application (82 with methods) 84 Development 7 Impact 6 Comparison 4 Intervention 62 (74%) Frequency 10 (12%) Causal association 4 (5%) Diagnosis 4 (5%) Patient experience 2 (2%) Screening 2 (2%) Not applicable Study characteristics (continued) 9 No. of Rapid Reviews (n=100) 16

Results (continued) Duration of review 4% 23% Protocol mentioned 2% 73% 98% Databases Searched Grey literature search 24% 70% Scanning references 10% 51% Contacting authors 11% 22% Date limit Titles and abstracts screening 49% 34% 54% 20% SR method Methods characteristics 22% 57% 17% 0% 29% 68% 10% Quality appraisal 17% 46% 25% Data abstraction 40% 20% 35% Full-texts screening 6% 68% 17% 6% 67% 12% Language limit 10 12% 83% 40% Streamlined method 26% 60% Not reported 80% 100%

Results (continued) Approach Literature search 1 >1 database, published only 2 3 4 5 Search limit Screening Both date and One reviewer language Updating the literature search of a previous None One reviewer review, published only >1 database, grey Both date and One reviewer literature language >1 database, grey Either date or One reviewer literature language >1 database, grey Date One reviewer literature 5 most frequent rapid review approaches 11 Data abstraction Risk of bias appraisal One person abstracts, other verifies One person assesses, other verifies One reviewer Not performed One reviewer

Conclusions § Numerous rapid review approaches were identified § Little consistency exists in the field § Poor quality of reporting was observed § Prospective study comparing the results from rapid reviews to those obtained through systematic reviews is warranted. 12

Methods project 2: An international survey of rapid review producers Submitted to the J of Clin Epi

Objective and methods § Objective – To determine different rapid review approaches used by rapid review producers § Methods – – – 14 International survey of 63 organizations administered via Fluid. Survey pilot-tested prior to administration Reminders to non-respondents sent every 2 weeks Contacted through postal mail if no response $10 financial incentive from Amazon

Results 63 organizations contacted 22 did not respond 41 responses (65%) 15 Study flow figure of participants

Results (continued) Review Characteristics Duration of Review (weeks) Count (%) 1 -12 12 -26 26 -36 ≥ 52 Government Agencies & Health Ministries Commissioning Agency Healthcare Organizations, Hospitals & Community Healthcare Professionals Industry Government Agencies & Health Ministries Healthcare Professionals Target Audience Patients Researchers 16 Summary of rapid review characteristics 62 (70) 18 (20) 6 (7) 2 (2) 69 (78) 51 (58) 13 (15) 4 (5) 73 (83) 46 (52) 19 (22) 21 (24)

Results (continued) Rationale provided Decision-maker timeline Focused or brief question Lack of resources Increase efficiency (including timeliness) Broad understanding of an area Identify topics requiring a systematic review Update a systematic review Well-established intervention Evidence is unclear 17 Rationale for conducting rapid reviews Count (%) 57 (66) 8 (9) 5 (6) 4 (5) 2 (2) 1 (1)

Results (continued) Review Stage Most frequent streamlined approach Identifying relevant studies Used previous review(s) as a starting point 79 (92) Limitations on search strategy Limited review by date of publication 75 (88) Study selection Screening conducted by ONE reviewer only 68 (85) Data Abstraction Data abstraction performed by ONE reviewer only 67 (84) Quality (risk of bias) appraisal process Risk of bias assessed by ONE reviewer only 68 (86) Synthesis Narrative summary 75 (90) 18 Summary results of most frequently streamlined approaches Count (%)

Conclusions § Results are consistent with scoping review of rapid reviews § Rapid reviews usually conducted in 1 -12 weeks § Government agencies and health ministries are primary commissioners § Many different streamlined methods are being used. 19

Methods project 3: International consensus-building exercise regarding rapid reviews Submitted to the J of Clin Epi

Objective & methods § Objective – To conduct a consensus-building exercise to select a rapid review approach that will be prospectively tested in a diagnostic study § Methods – Editors, healthcare providers, researchers, policy-makers, and industry stakeholders (including participants of the CADTH rapid review summit) – Asked to rank the 5 most frequent rapid review approaches identified in our scoping review and survey using Fluid. Survey – Results presented to participants, followed by a facilitated discussion (online and in-person) and re-ranking exercise using Fluid. Survey 21

Results Online delphi In-person delphi 26 individuals contacted 130 individuals contacted 3 did not respond 40 did not respond 113 responses (72%) 22 Study flow figure of participants

Results (2) Rapid review approach Feasibility Timeliness Comprehensiveness Risk of Bias Approach 1 1 st 2 nd 4 th 1 st Approach 2 2 nd 1 st 5 th Approach 3 3 rd 2 nd Approach 4 4 th 2 nd 4 th Approach 5 5 th 1 st 3 rd *Ranked based on the distribution of "very" and "extremely" on the 7 -point Likert scale, except Risk of Bias was ranked on distribution of “not at all” and “very” 23 Summary of ranking results by approach

Conclusion § The highest ranked method was: Approach 1 ‒ Most feasible (72%, n=81 out of 113 responses) ‒ Lowest perceived risk of bias (12%, n=12 out of 103) ‒ 2 nd in timeliness (37%, n=38 out of 102) ‒ 5 th in comprehensiveness (5%, n=5 out of 100) § We will use the information from the consensus-building exercise to select the rapid review approach for a prospective study. 24

Ultimate goal of this research Rapid review definition (Shannon Kelly) Identify and characterize rapid review methods Identify 5 frequently used methods Diagnostic study to test a rapid review approach “We can give you results within 4 months, but the metaanalysis estimates will be inaccurate by 35%” 25

Proposed diagnostic study § Will use these results from our research program to inform a diagnostic study: − Index test: Rapid Review Approach − Reference standard: Systematic Review Collaboration between 3 Canadian Knowledge Synthesis Centers Targeting CIHR and PCORI § § Diagnostic Accuracy of Rapid reviews compared To Systematic reviews (DARTS) 26

Summary § This research program provides up-to-date information on rapid review methods reported in the literature, as well as stakeholder experiences and perceptions regarding rapid reviews § Poor quality of reporting was observed in the literature § Rapid reviews have many names and approaches and some methods might be more desirable than others § A prospective study comparing the results of rapid reviews to those obtained through systematic reviews is necessary. 27

Acknowledgements § Funding: Canadian Institutes of Health Research/Drug Safety and Effectiveness Network − Operating grant to update 2 systematic reviews, international survey, Delphi − New investigator award § Research team: Jesmin Antony, Wasifa Zarin § Co-investigators: Drs. Straus, Moher, Hutton, Sherifali 28

Questions? triccoa@smh. ca 30