PHacking A Practical Guide NeuroSkeptic neuroskepticgooglemail com http

P-Hacking: A Practical Guide @Neuro_Skeptic neuroskeptic@googlemail. com http: //blogs. discovermagazine. com/ neuroskeptic

The Nine Circles of Dante’s Inferno First Circle: Limbo Second Circle: Lust Third Circle: Gluttony Fourth Circle: Greed Fifth Circle: Anger Sixth Circle: Heresy Seventh Circle: Violence Eighth Circle: Fraud Ninth Circle: Treachery

The Nine Circles of Scientific Hell First Circle: Limbo Second Circle: Overselling Third Circle: Post-Hoc Storytelling Fourth Circle: P-Value Fishing Fifth Circle: Creative Use of Outliers Sixth Circle: Plagiarism Seventh Circle: Non-Publication of Data Eighth Circle: Partial Publication of Data Ninth Circle: Inventing Data

P-Fishing Fourth Circle: P-Value Fishing “Those who tried every statistical test in the book until they got a p value less than 0. 05 find themselves here, an enormous lake of murky water. Sinners sit on boats and must fish for their food. Fortunately, they have a huge selection of different fishing rods and nets (brand-names include Bayes, Student, Spearman and many more). Unfortunately, only one in 20 fish are edible, so they are constantly hungry. ”

P-Fishing • Also known as… ▫ ▫ ▫ ▫ Questionable Research Practices (QRPs)) Torturing the data P-Hacking Outcome reporting bias Undisclosed flexibility Researcher Degrees of Freedom …and more.

P-Hacking Works! • Collect some data

P-Hacking Works! • Collect some data • Try many statistical tests on the same data

P-Hacking Works! • Collect some data • Try many statistical tests on the same data • Or try many variants of the same data (e. g. removing ‘outliers’. )

P-Hacking Works! • Collect some data • Try many statistical tests on the same data • Or try many variants of the same data (e. g. removing ‘outliers’. ) • Or try looking at different variables within the dataset

P-Hacking Works! • Collect some data • Try many statistical tests on the same data • Or try many variants of the same data (e. g. removing ‘outliers’. ) • Or try looking at different variables within the dataset • Report the analyses that give the most favourable results (usually the lowest pvalues).

Why Hacking Is So Useful • There are many choices (‘researcher degrees of freedom’) in data analysis. • For example, in a simple task-based f. MRI data analysis, Joshua Carp found 7000 combinations of parameters (very conservative). Carp, J. (2012). On the plurality of (methodological) worlds: estimating the analytic flexibility of f. MRI experiments Frontiers in Neuroscience

How Not To Get Caught • There are many tools, but perhaps the most intuitive: the p curve Try it now! http: //www. p-curve. com/app 2/ Simonsohn, U. Nelson, L. D. Simmons, J. P. (2013). P-curve: a key to the file-drawer. Journal of Exp. Psychol General

How It Can Be Stopped Smulders YM (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology Chambers CD (2013). Registered Reports: a new publishing initiative at Cortex

Happy hacking! @Neuro_Skeptic neuroskeptic@googlemail. com http: //blogs. discovermagazine. com/neuroskeptic