Pubmed vs FullText Query Performance in Systematic Reviews

Pubmed vs. Full-Text Query Performance in Systematic Reviews : Application to Non-inferiority Clinical Trials André Nguyen Van Nhieu 1, 2, Katet Moez 1, Michel Nougairede 2, Xavier Duval 4, Michaël Schwarzinger 1 ATIP-AVENIR Inserm “Modélisation, Aide à la Décision, et Coût-Efficacité en Maladie Infectieuses”, U 738, Université Denis Diderot, Paris, France; 2 Département de Médecine Générale, Université Denis Diderot, Paris, France 3 Inserm U 738, Université Denis Diderot, Paris, France; 4 Inserm CIC 007, AP-HP, Hôpital Universitaire Bichat, Paris, France 1 1

I declare no conflicts of interest 2

Introduction (1) o Systematic review = extensive research of appropriate publications in the literature o Usually performed through Pubmed using key-words o Methodology appropriate when keywords in title/abstract and Me. SH 3

Introduction (2) o Comparison of Pubmed VS FULL-TEXT o Application: Non-inferiority trials in infectious diseases 1 o According to Piaggio CONSORT statement JAMA 2006: improving quality of reporting Non-inferiority trials o Hypothesis : Pubmed as sensitive as FULL-TEXT 4

Objective o To compare the performance of 2 query strategies to identify non-inferiority trials with mortality as a primary outcome in infectious diseases: n Pubmed n Full-text using the search engine of each journal 5

Methods (1) o Original articles published in 2001 -2012 o In : n Generalist journals : N Engl J Med, Lancet, JAMA, Ann Intern Med, BMJ, Arch Intern Med n Specialist journals : Lancet Infectious Disease, Clinical Infectious Disease, Journal of Infectious Disease, AIDS, Vaccine, Pediatrics, Plos. Med 6

Methods (2) 7

8

9

Methods (3) o Using : n PUBMED : keywords : [randomi* AND (non? inferior* OR not inferior OR is inferior OR was inferior) AND (surviv* OR alive OR mortality OR death* OR fatal*)] + indexation (randomized controlled trial, Kaplan-Meier method, survival, mortality, death) n FULL-TEXT method with the same keywords adapted according to each search engine 10

Methods (4) o Definitions : n True positive (TP): Non-inferiority trial (NIT) with mortality as primary outcome found by one or the other query strategy n False positive (FP): Original article but it is not a NIT or a NIT with primary outcome ≠ mortality n False negative (FN): NIT fount by one query search but not the other n Positive predictive value (PPV): probability to find a TP among all Positive of a query strategy 11

Results (1) Flowchart 256 articles in infectious diseases identified in PUBMED and FULL-TEXT : 81 in common and 175 more with FULL-TEXT 184 Non-inferiority trials Articles excluded(N=72) : - 8 Reviews - 3 Observational studies - 17 Pooled analysis - 11 SUP RCT phase 2 - 23 SUP RCT phase 3 - 4 equivalence trials - 2 secondary analysis - 4 follow-up of NIT Articles excluded (n= 28): - 28 NIT with outcome different from mortality True positive False positive 156 Non-inferiority with mortality in the outcome Articles excluded (n=101): - 88 articles with survival as a secondary outcome - 13 articles with per protocol analysis 55 Non-inferiority with mortality as a primary outcome = TP 12

Results (2) FULL-TEXT PUBMED Sensitivity Positive Predictive Value 55/55 (100%) 19/55 (34, 5%) False negative 55/256 (21, 5%) 19/45 (42, 2%) 0/55 (0%) 36/55 (65, 5%) 13

Results (3) o Subgroup analysis : n Sensitivity not different between generalist / specialist journals (p=0, 14) n Sensitivity FULL-TEXT & Pubmed not different <2008 / >2008 (p=0, 07) 14

Limits of FULL-TEXT method 1) Repetition in each journal search engine 2) Specificity of each journal search engine : learning curve 3) Access to journals for GP is not free 4) Time consuming 15

Conclusion (1) o FULL-TEXT is 100% sensitive o Pubmed detects only 34, 5% of noninferiority trials in infectious diseases o High rate of false positive in FULL-TEXT o However, False Positive easily identified and excluded through reading 16

Conclusion (2) o Testing the Full-text method in other medical fields or repeating in a few years (improving Pubmed? ) o Improvement to be made with CONSORT statement o Adding Non-inferiority in Me. Sh terms 17

o THANK YOU 18

Bibliography o Le Henanff et al. / RAVAUD Quality of Reporting of Noninferiority and Equivalence Randomized Trial JAMA 2010 o Piaggio et al. Reporting of Noninferiority and Equivalence Randomized Trials JAMA 2006 and Extension of the CONSORT 2010 Statement JAMA 2012