bias necessary to invalidate the inference 1 Threshold

} % bias necessary to invalidate the inference { 1

Threshold as Point of Indifference with Respect to Evidence Concerns about Bias Evidence Threshold Return on Investment 2

Ordered Thresholds Relative to Transaction Costs Definition of threshold: the point at which evidence from a study would make one indifferent to policy choices 1. Changing beliefs, without a corresponding change in action. 2. Changing action for an individual (or family) 3. Increasing investments in an existing program. 4. Initial investment in a pilot program where none exists. 5. Dismantling an existing program and replacing it with a new program. 3

No Single Level of Reliability “An integrated collective epistemic enterprise requires the general aim of truth to be supplemented by a more specific consensus on the benefits of getting it right and the costs of getting it wrong in each case” (Wilholt, page 252, emphasis added). 4

Proposed Thresholds and Return on Investment just rule of thumb! Low Return Medium Return High Return Low allocation H 0 δ#=. 2 (light intervention) H 0 δ#=. 1 H 0 δ#=0 (antibiotic) Medium Allocation H 0 δ#=. 3 H 0 δ#=. 2 (Kindergarten Retention) H 0 δ#=. 1 High allocation H 0 δ#=. 4 H 0 δ#=. 3 (charter schools) H 0 δ#=. 2 (Surgery) δ# is the threshold effect size for making an inference 5

Threshold values for Open Court Effect on Achievement % Bias to Change Inference by Threshold Values Bias to Remove to. Invalidate the. Inference. 1 0. 9 0. 8 Specified Threshold % Bias 0. 7 0. 6 Statistical significance 0. 5 Estimated Effect 0. 4 0. 3 0. 2 0. 1 0 0 0. 1 0. 2 0. 3 0. 4 Threshold Value 0. 5 0. 6 itcv Impact Threshold for a Confounding Variable 6

Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489 -500. 7

Minimally Clinical Important Difference (MCID) Mc. Glothlin, A. E. , & Lewis, R. J. (2014). Minimal clinically important difference: defining what really matters to patients. Jama, 312(13), 1342 -1343. 8