Standing on the shoulders of giants Why your

Standing on the shoulders of giants: Why your literature review should be systematic Dorothy V. M. Bishop Professor of Developmental Neuropsychology University of Oxford @deevybee

Why did you pick your research topic? • Saw a great talk • Read a great paper • Supervisor/PI steered me • Funding linked to specific topic • Tractable problem • Personal reasons, e. g. relative affected by condition In most cases, you believe you are doing cumulative research – i. e. building on what has gone before

Structure of talk • Problem 1: Waste from unnecessary research • Problem 2: Trusting someone else’s literature review • Why you should do a systematic review before embarking on your own study (even if your PI doesn’t want you to) • Why a systematic review is much harder than you think • Some pointers on how to do a systematic review or meta-analysis

Wasting time and resources by ignoring prior literature

Unnecessary research Potential implications include ethically unjustifiable trials, wasted resources, incorrect conclusions, and unnecessary risks for trial participants. (Robinson & Goodman, 2011)

A common tragedy for ECRs: Trying to build on previous literature that is wrong What do I mean by wrong?

Very common in science

Confirmation bias • Cherry-picking may not be deliberate • We find it much easier to process and remember information that agrees with our viewpoint

A personal example: suppressed memory of one’s own research, when it does not fit Twin studies of SLI Twin concordance points to genetic influence when MZ > DZ Lewis & Thompson, 1992 Bishop et al, 1995 Tomblin & Buckwalter, 1998 Hayiou-Thomas et al, 2005 10 probandwise concordance: same-sex twins MZ DZ. 86. 48. 70. 46. 96. 69. 36. 33

A personal example: suppressed memory of one’s own research, when it does not fit Twin studies of SLI Twin concordance points to genetic influence when MZ > DZ Lewis & Thompson, 1992 Bishop et al, 1995 Tomblin & Buckwalter, 1998 Hayiou-Thomas et al, 2005 11 probandwise concordance: same-sex twins MZ DZ. 86. 48. 70. 46. 96. 69. 36. 33 I failed to mention this in talks for several years – I literally forgot about it!

Example of cherry-picking Example: Study published in 2013 “Regardless of etiology, cerebellar neuropathology commonly occurs in autistic individuals. Cerebellar hypoplasia and reduced cerebellar Purkinje cell numbers are the most consistent neuropathologies linked to autism [8, 9, 10, 11, 12, 13]. MRI studies report that autistic children have smaller cerebellar vermal volume in comparison to typically developing children [14]. ”

Paltering “Regardless of etiology, cerebellar neuropathology commonly occurs in autistic individuals. Cerebellar hypoplasia and reduced cerebellar Purkinje cell numbers are the most consistent neuropathologies linked to autism [8, 9, 10, 11, 12, 13]. MRI studies report that autistic children have smaller cerebellar vermal volume in comparison to typically developing children [14]. ” Mostly reviews

Example of cherry-picking Example: Study published in 2013 “Regardless of etiology, cerebellar neuropathology commonly occurs in autistic individuals. Cerebellar hypoplasia and reduced cerebellar Purkinje cell numbers are the most consistent neuropathologies linked to autism [8, 9, 10, 11, 12, 13]. MRI studies report that autistic children have smaller cerebellar vermal volume in comparison to typically developing children [14]. ”

Meta-analysis: Traut et al (2018) https: //doi. org/10. 1016/j. biopsych. 2017. 09. 029 Standardized mean difference is +ve when cerebellar volume is greater in ASD Webb et al did find area of vermis smaller in ASD after covarying cerebellum size

Paltering consistent with positive selfappraisal https: //hbr. org/2016/10/theres-a-word-for-using-truthful-facts-to-deceive-paltering

Ethically, is paltering (e. g. implying a review is an empirical paper, or citing inconsistent evidence as if consistent) better or worse than omitting evidence? See: Rogers, T. , Zeckhauser, R. , Gino, F. , Norton, M. I. , & Schweitzer, M. E. (2016). Artful paltering: The risks and rewards of using truthful statements to mislead others. Journal of Personality and Social Psychology, 112(3), 456 -473.

Inheritance of bias • When we read a peer-reviewed paper, we tend to trust the citations that back up a point • When we come to write our own paper, we cite the same materials • A good scientist won’t cite papers without reading them, but even this won’t save you from bias – you inherit it from prior papers • If prior papers only cite materials agreeing with a viewpoint, that viewpoint gets entrenched • You won’t know – unless you explicitly search – that there are other papers that give a different picture

Darwinian processes in survival of ideas “Examples of memes are tunes, ideas, catchphrases, clothes fashions, ways of making pots or of building arches. Just as genes propagate themselves in the gene pool by leaping from body to body via sperms or eggs, so memes propagate themselves in the meme pool by leaping from brain to brain via a process which, in the broad sense, can be called imitation. ” R. Dawkins

Successful meme • Easy to understand, remember, and communicate to others: • May be influenced by whether confers advantage to the person communicating • Survival does not depend on whether they are useful, true, or potentially harmful

A meme that is very close to home…. Experimenter bias effect Rosenthal and Fode (1963) • Two groups of students tested rats • Students wrongly informed either that the rats were specially bred to be "maze dull" or "maze bright. ” • In reality, all rats were standard lab rats, and were randomly assigned to the "dull" and "bright" conditions. • Rats labeled as "bright" learned the mazes more quickly than those labeled as "dull. "

But! The literature on experimenter bias is …. . a) Affected by p-hacking b) Subject to selective citation of studies finding an effect Barber, T. X. , & Silver, M. J. (1968). Psychological Bulletin, 70, 1 -29. http: //dx. doi. org/10. 1037/h 0026724 Critically analyzes 31 studies which attempted to demonstrate that Es' expectancies and desires significantly affect the experimental outcome (E bias effect). The majority of studies do not clearly demonstrate the effect. Many of these studies are criticized for inadequacies in the analysis of results, e. g. , failure to perform an overall statistical analysis to exclude the null hypothesis and failure to avoid probability pyramiding when postmortem tests are performed.

Motto of the Royal Society: “Take nobody’s word for it”

The solution Always start with a systematic review* • Systematic review • Collecting and summarise all empirical evidence that fits pre-specified eligibility criteria to address a specific question *If you are lucky, one will have been done, but odds are you may need to do it yourself ** If you are very lucky and working in a very new area, may be no prior literature, bit need to check

Guidelines for systematic review • Clearly stated objectives with pre-defined eligibility criteria for studies • Explicit, reproducible methodology • A systematic search that attempts to identify all studies • Assessment of the validity of the findings of the included studies (e. g. risk of bias) • Systematic presentation, and synthesis, of the characteristics and findings of the included studies Plan carefully, and document everything. • Consensus reporting guidelines for different study designs proposed by EQUATOR (http: //www. equator-network. org/) are a useful starting point. • PRISMA provides guidance on how to report a systematic review. See: http: //www. ccace. ed. ac. uk/research/softwareresources/systematic-reviews-and-meta-analyses

Example of how a systematic review can challenge accepted orthodoxy • In social science, wide interest in causes of social gradients in health. How far is this caused by differences in material factors, such as poverty, or social factors? • Research on primates allows some experimental control • One study that is frequently cited manipulation of social status produced five-fold increase in atherosclerosis “The effects of manipulating social status were dramatic. Dominants who became subordinate had a fivefold excess of coronary plaques compared with animals who remained dominant. ” (Brunner E. (1997) Socioeconomic determinants of health: stress and the biology of inequality. Br Med J 314: 1472. )

Petticrew and Davey-Smith did a systematic review…. . ‘Overall, non-human primate studies present only limited evidence for an association between social status and coronary artery disease. Despite this, there is selective citation of individual non-human primate studies in reviews and commentaries relating to human disease aetiology’(p. e 27937).

Petticrew and Davey-Smith also found that the ‘five-fold increase’ claim was based on small groups with NS difference on unadjusted data Friday, 20 July 2018 Standing on the shoulders of giants, or slithering around on jellyfish: Why reviews need to be systematic

Barriers to doing a systematic review • Not seen as valuable by PIs or by funders • Time-consuming • Many papers need reading and re-reading to extract key information (or discover it is not there) • Reluctance of authors to share data • If your results challenge the status quo, you may become unpopular Chalmers, I. (2005). Academia’s failure to support systematic reviews (letter). British Medical Journal 365, 469. https: //doi. org/10. 1016/S 01406736(05)17854 -4

Advantages of doing a systematic review • Less likely to waste time pursuing false leads • Learn a huge amount about methods in the research area y t li • Seeing what others do wrong can improve your own research • Learn whose work is trustworthy • Gain reputational advantage f i t n • Get a publication q ic e i c S a u • Opportunities to collaborate – or to enlist students in the project

Meta-analysis • Not a complicated technique: it is simply an average of study effect sizes, weighted by a measure of study precision. • Using the principal of aggregation, assume that idiosyncrasies and unreliability associated with each small individual study are averaged out, and a more precise – presumably more accurate – estimate is rendered in the aggregate. • Also can measure variability (termed heterogeneity in metaanalysis lingo) in effect sizes and test for moderators that explain this variability Corker, K. (2018) https: //psyarxiv. com/6 gcnm/

Meta-analysis: challenges Garbage in, garbage out! If you average together good and poor studies, you end up with a mess Solution: evaluate study quality Publication bias and p-hacking Solution: Search for unpublished studies Consider distribution of p-values or other stats to detect bias – meta-regression looks at association between effect size and study precision Heterogeneity / one-off studies Solution: Consider impact of moderators on results

Meta-analysis Metagear package: Automates aspects of data extraction/study selection Statistical analysis in R: metafor package http: //www. metafor-project. org/doku. php Metafor package: for analyses. See Lakens presentation

Professor Dorothy Bishop Wellcome Trust Principal Research Fellow, Oxford Study of Children’s Communication Impairments, Department of Experimental Psychology, South Parks Road, Oxford, OX 1 3 UD. @deevybee 34