Positive Controls Positive controls help ensure interpretable results
Positive Controls • Positive controls help ensure interpretable results. • Training: Students must replicate a standard finding before conducting independent research • Procedural Competence and Sensitivity: Positive controls are included in every experiment, demonstrating correct procedure and sufficient data. • Outlier Analysis: We assess each sample on positive-control responses, providing an independent screen for invalid responses. Positive controls are rare in psychology; Let’s change that!
Positive Controls In Psychology Studies Effect of Organic Food on Moral Judgements: Original Study (Eskine, 2013, Red) and Replications (Blue) Undergraduate research error or better estimate? Positive Control Green line: expected effect size. Probably not researcher error! Positive Control: Retrospective Gambler’s Fallacy (Oppenheimer & Monin, 2009) See a gambler roll 3 sixes or 2 sixes and a three. How many previous rolls? More rolls estimated in all-six scenario: dmedian = 0. 61 in Many Labs 1 (Klein et al. , 2014) Conducted at end of each replication study; Independent group assignment Moery & Calin-Jageman, 2016, SPPS
Positive Control Selection • Same research domain • Well-established effect size (RRRs and Many. Labs) • Similar in magnitude to the predicted effect or a range of positive control effects can be tested (small, medium, large) • Easy/short to administer • Sensitive to procedural issues thought to be important for the predicted effect • Think about what you’re most worried about going wrong in your study (nonnaïve participants, careless responding, lack of privacy, etc. ) • Select a positive control that is spoiled when this goes wrong • Bonus: Assessable at the individual level
Making Positive Controls Routine Let’s work on: • Library of off-the-shelf Positive Controls for different domains of psychology (OSF) • Sensitivity studies – some initial investigations of sensitivity to potential positive controls to procedural issues • Paper Benefits: • More interpretable results, less waste • More published replication research • Accumulation of data on PC effects to identify boundary conditions and moderators • Accumulation of data on what procedural issues are truly important in different research domains • Training standards and screens for bias
- Slides: 4