Basics of f MRI Inference Douglas N Greve

Basics of f. MRI Inference Douglas N. Greve

Overview • Inference • False Positives and False Negatives • Problem of Multiple Comparisons • Bonferroni Correction • Cluster Correction (voxel-wise threshold) • False Discovery Rate • Selection Bias

Statistical Inference • Can your conclusions be extended to data you have not seen? – Subjects, Time Points, Groups, Scanners • Or are your results the product of a chance occurrence that is unlikely to be repeated? • Generalizability, Repeatability, Reproducibility, Predictability • Uncertainty • Beyond good Experimental Design Group Population (All members) Hundreds? Thousands? Billions? Sample 18 Subjects 3

Truth Table Reality Conclusion Effect Is Not Effect Is Present (Neg) Present (Pos) True Negative False Positive Effect Is Not Present (Neg) Effect Is False Negative True Positive Present (Pos) 4

Error Rate Conclusion Reality Effect Is Not Present (Neg) Effect Is Present (Pos) True Negative False Positive TNR=1 -a FPR = a False Negative True Positive FNR = b TPR = 1 -b (Power) False Positive Rate (FPR) – probability that you declare an effect to be present when there is no effect False Negative Rate (FNR) - probability that you declare no effect to be present when there is an effect 5

How Do You Draw Conclusions? Protocol: reduce all your data to one number (the “test statistic” T). • If T is greater than some threshold (q) then conclude that an effect is present (ie, a positive) • Otherwise conclude that an effect is not present (ie, a negative). Every protocol has some FPR and some FNR, though it is not always easy to figure out! 6

Noise Causes Uncertainty Voxel 1 Voxel 2 7

GLM Inference T=8 T=1 8

Example Protocol • Collect data • Motion Correct • Smooth by 5 mm FWHM • Extract Voxel 1 (throw away rest of data) • Compute Mean and Std. Dev of ON time points • Compute Mean and Std. Dev of OFF time points • Compute test statistic T • If T > 3. 41, Conclude that the voxel is active Test Statistic (T) is the t-ratio Threshold (q) is 3. 41 What is the FPR (a) and FNR (b) for this protocol? 9

Example Protocol: False Positive Rate • “NULL” Distribution Student’s t-Distribution • p-value is area under curve to the right of T • For T = 3. 4, FPR = p =. 01 • For T=8, FPR = p = 10 -11 • For T=1, FPR = p = 0. 1 Student’s t Distribution • Assumptions: • Gaussian noise • Independent noise • Homoskedastic (equal variances) • Violation of assumptions change FPR=area under curve to the right of line (p-value) 10

Example Protocol: False Negative Rate • Need to know what the effect size is • Previous data • Guess • Power Analysis • Grants require a power analysis! 11

Trade Off of Error Rates FPR=. 10 FPR=. 01 • Inverse relationship between error rates • As False Positives (a) are reduced, the False Negatives (b) increase • Increase sample size decreases b, does not affect a • Which Error is more important? Depends. . • Science? FPR=. 05 ish, TNR<0. 2 • Pre-operative surgery? FPR=10 -7 12

What conclusions to draw from this? • Brain is activated? • Visual Cortex? • Auditory Cortex? • False Positive Rate? Need a protocol! 13

Possible Protocol • First Level Analysis • Compute t-ratio for each voxel • Compute p-value for each voxel • If any brain voxel has p <. 01, declare a positive • Same as • Test Statistic: T = max(Ti) • Threshold: q=3. 4 What is the False Positive Rate for this protocol? 14

What does p<. 01 mean? Rand(0, 1) 100 x 100 10, 000 vox p < 0. 1 1000 vox p < 0. 01 100 vox p < 0. 001 10 vox • p<. 01 means one expects 1% of voxels will be active purely by chance • Protocol gives a False Positive any time even a single voxel has p<. 01 • What is the probability that at least one voxel has p<. 01? 15

The “Problem of Multiple Comparisons” N = 10, 000 a. Vox • a. Vox = voxel-wise threshold (p< a. Vox) • a. FWE = Protocol False Positive Rate (FWE = Family-wise Error) • N = Number of voxels (“Search Space”) a. Vox =. 10 a. Vox =. 01 a. Vox =10 -7 a. FWE 0. 00001 0. 095 0. 0001 0. 632 0. 001 1. 000 0. 01 1. 000 16

Bonferroni Correction Compute Voxel-wise threshold needed to achieve a desired Family-wise FPR. To achieve a. FWE = 0. 01 with N = 10, 000 voxels Need a. Vox = 0. 000001 (10 -6) 17

Search Space • Set of voxels over which positives are searched • Severity of correction increases with size of search space (regardless of method) • Reduce Search Space • Reduce the area to a ROI (eg, superior temp gyrus) • Increase voxel size (cover same volume with fewer voxels) • Spatial Smoothing 18

Spatial Smoothing • Spatially convolve image with Gaussian kernel. • Kernel sums to 1 • Full-Width/Half-max: FWHM = s/sqrt(log(256)) s = standard deviation of the Gaussian 0 FWHM 5 FWHM 10 FWHM Full-Width/Half-max Full Max 2 mm FWHM Half Max 5 mm FWHM Smoothing causes irreversible loss of information (resolution) 10 mm FWHM

Spatial Smoothing causes irreversible loss of information (resolution), similar to increasing voxel size. 0 mm 5 mm 10 mm Smoothing 1 mm Increased Voxel Size 4 mm 8 mm

Resel • Pixel = picture element • Voxel = volume element • Resel = resolution element (depends on smoothing level) Resel = (FWHM)3 for volumes Resel = (FWHM)2 for surfaces If FWHM>Voxel Size, fewer Resels than Voxels. Correct based on the number of Resels instead of number of voxels (math is more complicated, need Random Field Theory) Bonferroni

Clusters a. Vox =. 10 a. Vox =. 01 a. Vox =10 -7 • True signal tends to be clustered • False Positives tend to be randomly distributed in space • Cluster – set of spatially contiguous voxels that are above a given threshold.

Cluster-wise Correction • Cluster – set of spatially contiguous voxels that are above a given threshold. • Protocol • Perform 1 st level analysis. • Threshold volume at a. Vox • Find clusters. • If Cluster Size > Threshold (q), Declare a Positive • Test Statistic: Cluster Size • What is the FPR (a. FWE) for this protocol?

Random Field Theory a. FWE = f(a. Vox, N, FWHM, Cluster. Size) p=. 05

Smoothing increases size of random clusters FWHM 0 Z Z>2. 3 p<. 01 FWHM 2 FWHM 4 FWHM 6

Cluster Images Sig Map p. Vox <. 001 Cluster Map p. Cluster <. 05 Some small clusters do not “survive”

Cluster Table MNI 305 Size Cluster Atlas Cluster X Y Z (mm 3) p-value Location 1 40 -67 -11 41368 ~0 Right Lateral Occipital 2 3 4 -40 -85 -13 51184 -6 17 -50 7 ~0 Left Lateral Occipital 45 2784. 00026 Left Superior Frontal 23 3768. 00002 Left Precentral R L Radiological Orientation ROI Atlas

Cluster Data Extraction • Spatial average over cluster of each subject’s contrast • Can correlate with other measures (age, test score, etc) • Be careful of “Selection Bias” (“Voodoo Correlations”)

Cluster Correction Summary • Cluster – set of supra-threshold voxels (size) • Critical Size Threshold given by Random Field Theory • Search Space • Voxel-wise threshold (arbitrary) • FWHM (smoothing level) • Assumptions on each • Loose small clusters (False Negatives)

False Discovery Rate (FDR) p < 0. 1 1000 vox p < 0. 01 100 vox p < 0. 001 10 vox • Given the voxel-wise threshold, know expected number of False Positives • If there are more Positives than this, then some of them must be True Positives

False Discovery Rate (FDR) • Number of False Positives = N*a. Vox • Total Number of Positives = Count from image • a. Vox = f(FDR, N, Data)

False Discovery Rate (FDR) • FDR =. 05 means that 5% of Positives are False Positives • Which 5%, no one knows • How to interpret? FDR =. 05 a. Vox =. 0070 FDR =. 01 a. Vox =. 0070

False Discovery Rate (FDR) • FDR =. 05 means that 5% of Positives are False Positives • Which 5%, no one knows • How to interpret? FDR =. 05 a. Vox =. 0070 FDR =. 01 a. Vox =. 0070 Would you change your opinion of this blob if 50 of the voxels were False Positives?

False Discovery Rate (FDR) • FDR =. 05 means that 5% of Positives are False Positives • Which 5%, no one knows • How to interpret? FDR =. 05 a. Vox =. 0070 FDR =. 01 a. Vox =. 0070 Would you change your opinion of this blob if 50 of the voxels were False Positives?

False Discovery Rate Summary • False Discoveries • FDR does not control FPR (False Positive Rate) • Careful when interpreting • Voxel-wise threshold is Data Dependent

Summary • Can your conclusions be extended to data you have not seen? • Truth Table: False Positives (a) and False Negatives (b) • Protocol – describes how you will draw conclusions • Problem of Multiple Comparisons (Family-wise Error) • Search Space, Search Space reduction • Larger voxels (less resolution) • Smoothing (Resels) • Bonferroni Correction • Cluster Correction (voxel-wise threshold) • False Discovery Rate • Selection Bias

37