The False Discovery Rate A New Approach to

The False Discovery Rate A New Approach to the Multiple Comparisons Problem Thomas Nichols Department of Biostatistics University of Michigan

f. MRI Multiple Comparisons Problem 1, 000 • 4 -Dimensional Data . . . – 1, 000 multivariate observations, each with 100, 000 elements – 100, 000 time series, each with 1, 000 observations 3 • Massively Univariate Approach – 100, 000 hypothesis tests • Massive MCP! 2 1

Solutions for Multiple Comparison Problem • A MCP Solution Must Control False Positives – How to measure multiple false positives? • Familywise Error Rate (FWER) – Chance of any false positives – Controlled by Bonferroni & Random Field Methods • False Discovery Rate (FDR) – Proportion of false positives among rejected tests

False Discovery Rate Illustration: Noise Signal+Noise

Control of Per Comparison Rate at 10% 11. 3% 12. 5% 10. 8% 11. 5% 10. 0% 10. 7% 11. 2% Percentage of Null Pixels that are False Positives 10. 2% 9. 5% Control of Familywise Error Rate at 10% FWE Occurrence of Familywise Error Control of False Discovery Rate at 10% 6. 7% 10. 4% 10. 5% 14. 9% 9. 3% 16. 2% 13. 8% 14. 0% Percentage of Activated Pixels that are False Positives 12. 2% 8. 7%

Benjamini & Hochberg Procedure p(i) i/V q/c(V) 0 • Reject all hypotheses corresponding to p(1), . . . , p(r). p-value p(i) i/V q/c(V) 1 • Select desired limit q on E(FDR) • Order p-values, p(1) p(2) . . . p(V) • Let r be largest i such that 0 JRSS-B (1995) 57: 289 -300 i/V 1

Benjamini & Hochberg Procedure • c(V) = 1 – Positive Regression Dependency on Subsets • Technical condition, special cases include – Independent data – Multivariate Normal with all positive correlations • Result by Benjamini & Yekutieli. • c(V) = i=1, . . . , V 1/i log(V)+0. 5772 – Arbitrary covariance structure

Benjamini & Hochberg: Varying Signal Extent p= Signal Intensity 3. 0 z= Signal Extent 1. 0 Noise Smoothness 3. 0 1

Benjamini & Hochberg: Varying Signal Extent p= Signal Intensity 3. 0 z= Signal Extent 2. 0 Noise Smoothness 3. 0 2

Benjamini & Hochberg: Varying Signal Extent p= Signal Intensity 3. 0 z= Signal Extent 3. 0 Noise Smoothness 3. 0 3

Benjamini & Hochberg: Varying Signal Extent p = 0. 000252 Signal Intensity 3. 0 z = 3. 48 Signal Extent 5. 0 Noise Smoothness 3. 0 4

Benjamini & Hochberg: Varying Signal Extent p = 0. 001628 Signal Intensity 3. 0 z = 2. 94 Signal Extent 9. 5 Noise Smoothness 3. 0 5

Benjamini & Hochberg: Varying Signal Extent p = 0. 007157 Signal Intensity 3. 0 z = 2. 45 Signal Extent 16. 5 Noise Smoothness 3. 0 6

Benjamini & Hochberg: Varying Signal Extent p = 0. 019274 Signal Intensity 3. 0 z = 2. 07 Signal Extent 25. 0 Noise Smoothness 3. 0 7

Benjamini & Hochberg: Properties • Adaptive – Larger the signal, the lower the threshold – Larger the signal, the more false positives • False positives constant as fraction of rejected tests • Not a problem with imaging’s sparse signals • Smoothness OK – Smoothing introduces positive correlations

FDR: Example • Verbal fluency data • 14 42 -second blocks • ABABAB. . . • A: Two syllable words presented aurally • B: Silence • Imaging parameters – 2 Tesla scanner, TR = 7 sec – 84 64 x 64 images of 3 x 3 mm voxels

FDR Example: Plot of FDR Inequality p(i) ( i/V ) ( q/c(V) )

FDR: Example FDR 0. 05 Indep/PRDS t 0 = 3. 8119 FDR 0. 05 Arbitrary Cov. t 0 = 5. 0747 FWER 0. 05 Bonferroni t 0 = 5. 485

FDR Software for SPM http: //www. sph. umich. edu/~nichols/FDR

FDR: Conclusions • False Discovery Rate – A new false positive metric • Benjamini & Hochberg FDR Method – Straightforward solution to f. NI MCP – Just one way of controlling FDR • New methods under development e. g. C. Genovese or J. Storey • Limitations – Arbitrary dependence result less sensitive Start Ill http: //www. sph. umich. edu/~nichols/FDR Prop

References • Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57: 289 --300. • Benjamini, Y, Yekutieli D (2002). The control of the false discovery rate under dependence. Annals of Statistics. To appear. • Genovese CR, Lazar N, Nichols TE (2002). Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. Neuro. Image, 15: 870 -878.

Positive Regression Dependency • Does f. MRI data exhibit total positive correlation? • Example – 160 scan experiment – Spatial autocorrelation of residuals – Single voxel with all others • Negative correlation exists!