Szapudis talk False Detection Rate Simultaneous hypothesis testing

Szapudi´s talk – False Detection Rate Simultaneous hypothesis testing - Setting the statistical significance Detections of Non-Gaussianity in CMB observations • Brief review of detections/methods used to set the statistical significance. • Application of FDR to individual statistical methods. • Application of FDR to a combination of statistical methods? . • What is the appropriate method for assessing the statistical significance of localized detections? .

CMB: Non-Gaussianity tests Real space Spherical Harmonic space Wavelet space Blind tests – The alternative to the null hypothesis is not specified.

Non-Gaussianity tests: Real Space Based on the temperature fluctuation observed at each pixel i, ΔT(i) Tests (detections): • Genus (Park 2003). • N-point correlation function (Eriksen et al. 2003, Eriksen et al. 2004). • Minkowski functionals and length of skeleton (Eriksen et al. 2004). • Extrema (Larson & Wandelt 2004) • 2 -point correlation function of maxima and minima (Tojeiro et al. 2005, Larson & Wandelt 2005) Courtesy of WMAP Science Team

Non-Gaussianity Tests: Spherical Harmonic Space Based on the complex coefficients Tests (detections): • Power spectrum distribution (Eriksen et al. 2003, Hansen et al. 2004). • Correlations between adjacent multipoles (Prunet et al. 2004).

Non-Gaussianity Tests: Wavelet Space Based on wavelet coefficients calculated at each pixel i, at a given scale R, wv(i, R). Tests (detections): • Kurtosis – Spherical Mexican Hat Wavelet (Vielva et al. 2003, Mukherjee & Wang 2004, Mc. Ewen et al. 2004, Liu & Zhang 2005, Cruz et al. 2006). • Skewness – Real Morlet Wavelet (Mc. Ewen et al. 2004, Liu & Zhang 2005). • Number and area and volume of spots – SMHW (Cruz et al. 2004, Cruz et al. 2006). • Higher Criticism – SMHW (Cayon et al. 2005, Cruz et al. 2006).

Testing the null hypothesis through Monte Carlo simulations Simulating a CMB map: At each pixel (i – θ, ψ) 1) Bl – Antenna Cl - Cosmology 2) Add Noise - dispersion given by simulated experiment. Mask – Galaxy plus point sources (zeros to the pixel in the mask). Monopole and dipole removal.

Testing the null hypothesis through Monte Carlo simulations Map ΔT(i) Test statistic: Over whole map or Different combinations of pixels (N-point corr. ) alm Test Statistic: Several multipoles or Several combinations of multipoles (bispectrum) Wavelet coeff. wv(i, R) at scale R Test Statistic: Over whole map at each scale and/or above/below threshold (spots)

Statistical Hypothesis Testing • Testing the null hypothesis with a single configuration (confidence level). • Simultaneous hypotheses testing : - Χ 2 Example (Mc. Ewen et al. 2004) SMHW Skewness and kurtosis – 24 statistics. Detection at the 99. 9% significance level. - Conservative significance level Based on marginal distribution of all statistics. Ex. Above 95. 3% significance level. - Hypothesis Test SMHW Kurtosis- scale 9 above 99% confidence level. Larson & Wandelt, astro-ph/0505046 (maximum risk of false detection at the same level as the claimed significance)

Statistical Hypothesis Testing (FDR) • Simultaneous Hypotheses testing – False Detection Rate - Control of the fraction of false discoveries (detections) over the total number of discoveries. - No assumption on the Gaussianity of the error distribution. - Correlations between statistics can be taken into account (? ). αx 100 % of discoveries may be mistakes.

False Discovery Rate – Ex. Wavelet Space 16 tests / statistical test No correlations, α=0. 05, detection scales=9, 8, 7 Correlations, α=0. 1, detection scales=9, 8, 7 No correlations, α=0. 1, detection scales=9, 8 Correlations, α=0. 2, detection scales=9 Figs from Cruz et al. 2006

False Discovery Rate – Ex. Wavelet Space Statistical Test No Correlations CN=1 CN=3. 38 (16 scales) α |scales (detect. ) α|scales (detect. ) Kurtosis α=0. 05 9, 8, 7 α=0. 1 9, 8, 7 Max α=0. 05 9, 10 α=0. 2 9, 10 Higher Criticism α=0. 1 9, 8 α=0. 2 9 Area above 3σ α=0. 05 8, 9 α=0. 2 8, 9 Area above 3. 5σ α=0. 05 9, 8 Area above 4σ α=0. 05 9, 8, 10 α=0. 2 9, 8, 10 Area above 4. 5σ α=0. 1 9, 8 α=0. 2 9 Acknowledgement – M. Cruz

What is the appropriate method for setting the statistical significance? • Only considering detections based on SMHW: 112 statistical tests. - Different statistical methods. - Several scales. • Is it possible to assess the statistical significance of all detections all together? • The pixels behind some of the detections are localized.