How Blind to do analysis a believable in
How Blind to do analysis a believable in Ice. Cube analysis (blind analysis can be one part of this process. . . ) Gary C. Hill University of Wisconsin, Madison Photograph: Forest Banks Gary C. Hill, MANTS, Sep 26 th, 2009
“If your experiment needs statistics, you ought to have done a better experiment. ” My translation: “Sometimes discoveries are obvious” Gary C. Hill, MANTS, Sep 26 th, 2009
Milagro • Views northern sky • 2600 m altitude • Water-Cerenkov EAS detector • Continuous operation • Separates gammas from hadrons Gary C. Hill, MANTS, Sep 26 th, 2009
Milagro sky map Cygnus Mrk 421 Crab Gary C. Hill, MANTS, Sep 26 th, 2009
Milagro map-of. Cygnus the Cygnus region Milagro region MGRO J 2019+37 Te. V J 2032+413 (HEGRA IACT) 3 EG J 2033+4118 Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
“real sources” “candidate sources” Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
der Kluge Hans Gary C. Hill, MANTS, Sep 26 th, 2009
What are our biases? • We unconsciously look for – the “right” result – a “significant” result – discovery – a “non-significant” result – upper limit – an explanation based on the detector • and this all depends on your prejudices! Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Medical trials “Double blind placebo controlled study” Split subjects into two groups Half given new treatment (active) Half given fake treatment (placebo) Neither Doctors nor patients know Results should be unbiased. . . active – placebo = effect strength often strong effect in placebo group due to expectation of a. Gary cure C. Hill, MANTS, Sep 26 , 2009 th
der Kluge Hans Gary C. Hill, MANTS, Sep 26 th, 2009
Point source analysis • Scrambled data provides a perfect estimator of background • Everything is in there, no need for background simulation • Significances are correct • Blind analysis can be done, so do it • If you a posteriori data-mine, verify with new data Gary C. Hill, MANTS, Sep 26 th, 2009
Following Rutherford’s advice! Esteban Roulet Gary C. Hill, MANTS, Sep 26 th, 2009
Esteban Roulet Gary C. Hill, MANTS, Sep 26 th, 2009
Esteban Roulet Gary C. Hill, MANTS, Sep 26 th, 2009
Esteban Roulet Gary C. Hill, MANTS, Sep 26 th, 2009
Esteban Roulet Gary C. Hill, MANTS, Sep 26 th, 2009
Diffuse analyses Atmospheric neutrinos “Reasonable” charm flux uncertainty Astrophysical diffuse source ~ E-2 Detector systematics Waxman-Bahcall upper bound Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Clean upgoing neutrino sample mostly atmospheric neutrinos – any point sources or extra high energy events? Gary C. Hill, MANTS, Sep 26 th, 2009
Diffuse searches are hard • Background must be well predicted (no “off-source” / “side-band” region for direct measurement!) • Backgrounds – atmospheric muons (angle) – atmospheric neutrinos (angle, energy) • pi, k decays • charm decays – detector systematics must be well understood Gary C. Hill, MANTS, Sep 26 th, 2009
Energy separation: Nch variable Blind analysis: Best limit setting power (model rejection potential) Nch>100 E-2 test flux Atmospheric neutrinos Blind region Gary C. Hill, MANTS, Sep 26 th, 2009
Atmospheric error band includes uncertainties on: 1. primary cosmic ray flux 2. simulation of interactions 3. Atmospheric charm 4. detector effects 5. effects of selection cuts 10 -6 test flux: 66 events 35% uncertainty (detector response) Atmospheric: 6 events Gary C. Hill, MANTS, Sep 26 th, 2009
Atmospheric neutrino uncertainties constrained by low energy events normalise nch 50 -100 10 -6 test flux: 66 events 35% uncertainty (detector response) Atmospheric: 6 events Gary C. Hill, MANTS, Sep 26 th, 2009
Atmospheric neutrino uncertainties constrained by low energy events normalise nch 50 -100 10 -6 test flux: 66 events 35% uncertainty (detector response) Atmospheric: 6 events Gary C. Hill, MANTS, Sep 26 th, 2009
Sensitivity: for bg=6. 1 events median event limit 5. 84 Flux limit : 5. 84/66. 7 x 10 -6 = 8. 8 x 10 -8 Sensitive energy range (90% events) 104. 2 Ge. V to 106. 4 Ge. V = 15. 8 Te. V to 2. 51 Pe. V Gary C. Hill, MANTS, Sep 26 th, 2009
Sensitivity: for bg=6. 1 events median event limit 5. 84 Flux limit : 5. 84/66. 7 x 10 -6 = 8. 8 x 10 -8 Gary C. Hill, MANTS, Sep 26 th, 2009
Unblind the data: 6 events observed event limit 5. 84 Flux limit : 5. 84/66. 1 x 10 -6 = 8. 8 x 10 -8 Gary C. Hill, MANTS, Sep 26 th, 2009
Unblind the data: 6 events observed event limit 5. 84 Flux limit : 5. 84/66. 1 x 10 -6 = 8. 8 x 10 -8 Gary C. Hill, MANTS, Sep 26 th, 2009
Probability of observable shape distribution {ni } given theory {μi} ? Product of Poisson probabilities over bins: m parameters of interest; k-m nuisance parameters Gary C. Hill, MANTS, Sep 26 th, 2009
Non-fitted data Gary C. Hill, MANTS, Sep 26 th, 2009
Fitted data Gary C. Hill, MANTS, Sep 26 th, 2009
99. 7% (3σ) acceptance region – upper limits on Charm and ET Number of E-2 Number of Charm On all acceptance plots the global best fit point is marked as a red square! Gary C. Hill, MANTS, Sep 26 th, 2009
Non-fitted data Gary C. Hill, MANTS, Sep 26 th, 2009
Fitted data Gary C. Hill, MANTS, Sep 26 th, 2009
99. 7% acceptance region – clear excess of events, ET=0 disfavoured but still large allowed region for Charm… Number of E-2 Number of Charm Gary C. Hill, MANTS, Sep 26 th, 2009
• Need to avoid experimenters bias • Unbiased cut selections do this • Likelihood analysis needs no cuts and so avoids cut bias • Systematics can still be a big problem. . . • Blindness is no protection Gary C. Hill, MANTS, Sep 26 th, 2009
Rolling it up • Do whatever you can to avoid experimenters’ bias. . . • Do whatever you can to convince your audience that you have avoided experimenters’ bias. . . Gary C. Hill, MANTS, Sep 26 th, 2009
Analysis procedures in Ice. Cube • Develop analysis proposal in working group – Point source with scrambled data – Diffuse with high-energy tail hidden, or using only 10% of data • Working group review and approval • Collaboration review via analysis call • Permission to unblind Gary C. Hill, MANTS, Sep 26 th, 2009
So you’ve designed your nice blind analysis, passed all your data challenges, then open up the signal region and find that things make no sense whatsoever? What do you do next? Let’s ask the experts. . . Gary C. Hill, MANTS, Sep 26 th, 2009
When something goes wrong? Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
Gary C. Hill, MANTS, Sep 26 th, 2009
- Slides: 52