How Many Scientists Fabricate and Falsify Research A
How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data Daniele Fanelli
“The million dollar question”…. Data source Government-confirmed cases in the US Frequency (%) of misconduct 0. 001 – 0. 01 (Marshall 2000, Steneck 2006) 0. 02 - 0. 2 (Claxton 2005) Manipulated images submitted to The Journal of Cell Biology 1 (Steneck 2006) Investigators disqualified by FDA data audits (1977 -1990) 2 (Glick 1992) Retractions due to misconduct, in Pub. Med -Only misconduct that has been discovered, and (presumably) proven to be intentional Ultimately, only scientists know about their own intentions!
Over the years, many surveys have asked scientists directly… …different things, in different ways… Form of misconduct Outcome Question “Since entering medical school have you…? ” “Have you participated in research involving […] during the last 10 years? ” “Indicate the number of […] members you have observed/experienced exhibiting […] within the last 5 years” “Fabricated data” Yes No “Modified research or experimental results to improve the outcome” “Failing to present data that contradict one's own previous research” “Seriously misleading interpretation of results” Never Sometimes Frequently 0 1 -5 >5 Results appeared inconclusive and dificult to compare
“Tricks” in the analysis • How many committed or observed X at least once • Only questions on fabrication, falsification, alteration and QRP that distort scientific knowledge. No plagiarism, professional misconduct etc… • Mixed questions were excluded p ù é ES=Log Question by question: effect size and weight ë (1 - p) û 1 W= 2 = n p(1 - p) SE • No other measure of study quality (it’s controversial) – Included all eligible studies that specified their methods – Entered methodological factors in inverse variance weighted regression
The search… "research misconduct" OR "research integrity" OR "research malpractice" OR "scientific fraud" OR "fabrication, falsification" OR "falsification, fabrication" 42 literature databases, 14 journals, 8 grey literature db, 2 internet scientific search engines, and references lists Potentially relevant studies obtained from electronic search (n=3276) Studies retrieved for examination of full text (n=69) Studies included in review (n=21) Studies excluded because were not surveys on research misconduct (n=3207) Studies excluded for one of the following reasons (n=48): -Did not have any relevant or original data -Sample not exclusively composed of researchers -Misconduct not related to research (e. g. cheating on school projects) -Does not distinguish fabrication and falsification from other forms of misconduct not relevant to this review -Presents data only in format not usable in this review Studies excluded from meta-analysis because did not meet quality criteria (n=3) Studies included in metaanalysis (n=18)
Characteristics of studies • Conducted between 1986 -2005 • USA (15), UK (3), multinational (2), and Australia (1) • Medical/clinical (8), biomedical (6), multidisciplinary (6), economists (1) • In total 85 questions: – about fabrication, falsification, alteration, modification (meta-analysis) – Questionable research practices (systematic review only) (Full data set available soon in PLo. S ONE)
Scientists who admit fabrication, falsification, or alteration of results Scientists who know a colleague who fabricated, falsified, or altered results b= -0. 14± 0. 05 P=0. 006 1. 97% (N=7, 95%CI: 0. 86 -4. 45) If only asked “fabrication, falsification” 1. 06% (N=4, 95%CI: 0. 31 -3. 51) 14. 12% (N=12, 95% CI: 9. 91 -19. 72) If only asked “fabrication, falsification” 12. 34% (N=11, 95%CI: 8. 43 -17. 71)
Questionable Research Practices (e. g. “failing to publish data that contradicts one’s previous research” “dropping data points based on a gut feeling”)
What influences admission rates? Inverse variance-weighted regression Asking about self vs colleagues: Using “fabrication” or “falsification” vs “alteration” or “modification” Handed-out surveys vs mailed: + b±SE P -4. 53± 0. 81 <0. 0001 -1. 02± 0. 39 0. 0086 1. 17± 0. 4 0. 0032 82% of variance explained (N=15) Controlling for these factors, tested for differences between: Year USA / other Researcher / other Biomedical / other Social Sc. / other n. s. Medical / other b=0. 85± 0. 28 P= 0. 0022
Leave-one-out sensitivity analysis Scientists who admit fabrication, falsification, or alteration of results Martinson 2005 is outstanding, as conservative! Scientists who know a colleague who fabricated, falsified, or altered results
“Repairing misconduct” ID N cases Action taken % Tangney, 1987 78 Took some action to verify their suspicions of fraud or to remedy the situation 46 Rankin, 1997 31 (incl. Plag. ) In alleged cases of scientific misconduct a disciplinary action was taken by the dean 32. 4 Some authority was involved in a disciplinary action 20. 5 I interfered to prevent it from happening 28. 6 I reported it to a relevant person or organization 22. 4 Ranstam, 2000 Kattenbraker, 2007 Titus, 2008 49 33 Around of recalled cases had Confrontedhalf individual no action whatsoever taken against them 55. 5 Reported to supervisor 36. 4 Reported to Institutional Review Board 12. 1 Discussed with colleagues 36. 4 115 (incl. The suspected misconduct was reported by the survey Plag. ) respondent The suspected misconduct was reported by someone else 24. 4 33. 3
Summary of key findings • Data fabrication, falsification and alteration was – admitted on average by around 2% (1% - 4%) – directly observed by 14% (10% - 20%) • Questionable Research Practices were – admitted on average by up to 34% – directly observed by up to 72% • Overall admission rates (self-/non-self) were higher in – Non-self reports, questions not using “fabrication” or “falsification”, handed out questionnaires – Medical/clinical and related research
How Reliable Are These Numbers? Self-reports Conservative Unlike surveys for other criminal behaviour, Scientists always lose by admitting misconduct Non-self-reports Unclear - Risk of multiple reporting “Muhammad Ali” effect Unaware of all cases Unwilling to damage their field Regression analysis: -Medical research not robust to all sensitivity -Differences in methodology masked most effects -Non-significant effects not necessarily non-significant
Conclusions (tip of the iceberg) (this too) • On average, 2% of scientists admitted misconduct, and 34% QRP – Actual frequencies probably higher – Probably vary depending on field and many other factors, which meta-analysis currently cannot detect • Future surveys might benefit by – Focusing on correlates of misconduct – Common methodologies
- Slides: 14