Replication is Crucial for Theory and Innovation Ignazio

Replication is Crucial for Theory and Innovation Ignazio Ziano

s e r / t n e m t r e a r p e e h d a r e u r o a Y h c r ea

Reasoning behind assigned papers “False Positive Psychology” it is very easy to get bullshit results “Replication Recipe” this is how to replicate them “Making Replication Mainstream” replication is important “The Preregistration Revolution” preregistration is important

Why all this psychology? I am a consumer psychologist Psychology is at the forefront of the credibility revolution Other disciplines are lagging behind

Replication: what is it?

Replication – what is it? Can someone else repeat the same experiments and get the same results? Rough definition: Effect in same direction, p <. 05 Le. Bel et al 2018 for a better taxonomy Philosophically, replication is a falsificationist concept Statistically, replication is a frequentist concept But see: Bayes factors (Kass & Raftery 1995, JASA)

Replication: what is it NOT about Data problems leading to irreproducibility, weakness, non generalizability of results Imputations in international political economy: 50% significant not significant This is a big problem, but for another day

https: //en. wikipedia. org/wiki/Replication_crisis

Replication crisis - Psychology R: PP (Nosek et al 2015) 36/100 replicated effects both Statistically significant (p <. 05) same direction as original (36%) Lower in social psych (~25%) than cognitive psych (~55%)

Replication crisis - Psychology Many Labs 1: 10/13 successes (77%) Many Labs 2: 14/28 (50%) Many Labs 3: 3/10 (30%) Few “one-shot” large replications also failed to show effect Ego depletion Hagger et al 2016, N=2, 141 Professor priming O’Donnell et al 2018, N=6, 454 Stereotype threat Flore et al 2019, N=2, 064

How to identify replicable research in psychology Has this been replicated successfully? Curatescience. org contains a curated list of replications in Psychology Maintained by Etienne Le. Bel Overlap with many authors and theories in marketing, OB, management…

Curatescience. org

Curatescience. org

Curatescience. org

Replication crisis – Economics/Social Sciences EE-RP (Experimental Economics Replication Project): 11/18 (61%) RSSP: 13/21 (62%) – Science & Nature papers

Is social science replicable? 36/100 R: PP (Psychology) 11/18 EERP (Experimental Economics) 10/13 Many Labs 1 (Psychology) 14/28 Many Labs 2 (Psychology) 3/10 Many Labs 3 (Psychology) 13/21 SSRP (Science & Nature social science) 87/190 = 46% replication rate (54% false discovery rate) = Should we not do better?

Replication crisis – Cancer biology https: //www. nature. com/articles/nrd 3439 -c 1 https: //www. nature. com/articles/483531 a Amgen and Bayer Only 11 -25% of cancer biology papers replicated Amgen Bayer

Replication crisis – Cancer biology Couldn’t even agree on the materials… For now, 5 attempts. 2 successes, 1 failure, 2 inconclusive

Replication crisis: Medicine Ioannidis 2005, JAMA 20/34 (59%) highly cited studies (>1, 000 cites) were replicated 14/34 (41%) were not replicated

Replication crisis: Medicine Prasad et al 2013, MCP 146/363 (40%) tested medical practices did not work (not a representative sample)

Replication crisis – Neuroscience Boekel et al 2015 (neuroimaging papers) 1/17 (6%)

Is science in general replicable?

https: //www. nature. com/news/1 -500 scientists-lift-the-lid-on-reproducibility 1. 19970

Business disciplines

Business journals editors

Marketing – JCR, 2001

There are exceptions Gotteland’s editorial 2018, RAM

Psychology & Marketing Registration reports

Does preregistration make it easier to publish null results? JBR special issue

Journal of Business Ethics special issue Deadline: January 1 st 2019

Journal of Economic Psychology special issue Deadline: June 1 st 2018

The Desperate Need for Replication

Replication has to become a routine Everyone should try to directly replicate studies especially GM (honors thesis) students, 1 st year Ph. Ds, but also everyone else Journal editors should invite more replications Authors should ask, ask for replications to be published

Replication should get more recognition Replications are important! A good replication is as valuable as an original piece of research There is a real need for more direct replication rather than conceptual replication

What should we replicate? We should start with important papers that have a lot of impact e. g. , R: PP psych; Ioannidis 2005 investigated highly cited papers BUT ALSO newer, younger papers kill BS theories in the cradle, so people do not even think about building on them

Why is replication important for theory? It is very possible that some theories are false (e. g. , ego depletion; money priming) If we do not replicate them, we will not know

Why is replication important for innovation? It is very possible that some findings are bunk (e. g. , cancer biology replication rates; medical reversal) If we do not replicate them, we will not know The pace of innovation will slow because money, time, and effort will wastefully be spent on irreproducible results

The Desperate Need for Preregistration

Carney, Cuddy, and Yap 2010, Psych Science N = 42 Power poses increases testosterone & cortisol level, risk-taking Ranehill et al. 2015, Psych Science N = 200 No effect (all ps>. 16) Power poses do nothing for hormones & risk taking

http: //faculty. haas. berkeley. edu/dana_carney/pdf_ My%20 position%20 power%20 poses. pdf

A lot of subjective decisions Which are NOT the problem! The problem: transparency Did you really test a HP or were you fishing? How? Pre-registration

What is preregistration? ‘In pre-registration, researchers describe their hypotheses, methods, and analyses before a piece of research is conducted, in a way that can be externally verified’ Van ‘t Veer & Giner-Sorolla 2016 This means… write it down before you do it upload it on a 3 rd party website Where it is time-stamped Then run the study

Where& how can I preregister? Open Science Framework (osf. io) Aspredicted. org MANY other places E. g. , USA medical trials HAVE to be preregistered and are all at clinicaltrials. gov New publishing platform researchers. one

Aspredicted. org

Aspredicted. org

Aspredicted. org

Aspredicted. org

Why preregister? Transparency of your decision A priori vs. ad hoc Fine to ‘learn from the data’ if you specify what you planned and what you learned! (Prediction vs. postdiction) If you want to replicate other people’s work Make it clear that you are not out to get them!

Are people who do not preregister bad people? No. BUT preregistration makes your research more transparent Easier to publish null results ‘Look. We followed the recipe. We didn’t mess anything up. It just doesn’t work!’

Is preregistered research better? Not necessarily. Only more transparent, which leads to have less publication bias. Which means easier to criticize and spot mistakes Self-correcting science

HOWEVER… Preregistered research shows less evidence of publication bias

Clinicaltrials. gov Kaplan & Irvin 2018

Allen and Mehler (2018) psyarxiv. com/3 czyt List of journals offering RRs: cos. io/rr/

Preregistered studies are more reliable More symmetry in Egger’s plot Less evidence of publication bias!

Why is preregistration important for innovation Preregistration makes results more transparent Transparency makes results easier to evaluate Easier to evaluate whether you should trust a result or not Easier to discard bunk results (e. g. , cancer biology) and build on solid ones.

What about your discipline? Can you replicate? Can you preregister? Do you think there are replication problems?

Finance No problems?

Useful to have readers evaluate your research plan and possible changes even if you do not have a HP Preregistration does not hinder subjectivity in Qual, it just makes it explicit

Assignment #1 - Preregistration See Dropbox folder for materials 1. Compare preregistration with study 2. Do table with similarities and differences 3. Suggest improvements 4. Groups of 2 -3

Assignment #2 – Replication recipe 1. Pick an important empirical finding in your area 2. Build a replication recipe based on Brandt et al 2014 (modify if necessary) 3. Upload it on Dropbox (I will send email) 4. Individual

Download presentation: osf. io/afg 3 u (or on Dropbox) Ignazio. ziano@Grenoble-em. com Office F 710 Twitter: @ignaziano