http eilam weizmann ac ilImagesThe Higgs Mechanism swf

http: //eilam. weizmann. ac. il/Images/The. Higgs. Mechanism. swf 1 Eilam Gross, Higgs Statistics, Santander 2009

Statistical issues for Higgs Physics Eilam Gross, Weizmann Institute of Science 2 Eilam Gross, Higgs Statistics, Orsay 09

Bayesian 3 Eilam Gross, Higgs Statistics, Orsay 09 Modified Frequentist - CLs

CMS & ATLAS Higgs Prospects Bayesian CMS Gr י egory Schott SUSY 2009 4 Eilam Gross, Higgs Statistics, Orsay 09 Profile Likelihood ATLAS

Discovery vs Exclusion Higgs statistics is about testing one hypothesis against another hypothesis One hypothesis is the Standard Model with no Higgs Boson (H 0) Another hypothesis is the SM with a Higgs boson with a specific mass m. H (H 1) Rejecting the No-Higgs (H 0) hypothesis DISCOVERY Rejecting the Higgs hypothesis (H 1) EXCLUDING the Higgs at the 95% CL: Usually H 0 is referred to as the null hypothesis, but it depends on the context. 5 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definition: Test Statistic Given an hypothesis H 0 (Background only) one wants to test against an alternate hypothesis H 1 (Higgs with mass m. H) One (very good) way is to construct a test statistic Q and use it to accept or reject an hypothesis For a physicist the test statistic is part of the analysis model 6 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: p-Value A lot of it is about a language…. A jargon Discovery…. A deviation from the SM - from the background only hypothesis… When will one reject an hypothesis? p-value = probability that result is as or less compatible with the background only hypothesis Control region a (or size a) defines the criterion It is a custom to choose a=2. 87 10 -7 7 If result falls within the control region, i. e. p< a the BG only hypothesis is rejected A discovery Eilam Gross, Higgs Statistics, Orsay 09 The pdf of Q…. Control region Of size

From p-values to Gaussian significance It is a custom to express the pvalue as the significance associated to it, had the pdf were Gaussians Beware of 1 vs 2 -sided definitions! 8 Eilam Gross, Higgs Statistics, Orsay 09

LR motivation. The Neyman-Pearson Lemma When performing a hypothesis test between two simple hypotheses, H 0 and H 1, the Likelihood Ratio test, which rejects H 0 in favor of H 1, is the most powerful test of size for a threshold Define a test statistic Note: Likelihoods are functions of the data, even though we often not specify it explicitly 9 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Confidence Interval & Coverage Say you have a measurement mmeas of m. H with mtrue being the unknown true value of m. H Assume you know the pdf of p(mmeas|m. H) Given the measurement you deduce somehow (based on your statistical model) that there is a 90% Confidence interval [m 1, m 2]. . The correct statement: In an ensemble of experiments 90% of the obtained confidence intervals will contain the true value of m. H. The misconception; Given the data, the probability that there is a Higgs with a mass in. H the interval [m 1, m 2] is 90%. 10 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Confidence Interval & Coverage Confidence Level: A CL of 90% means that in an ensemble of experiments, each producing a confidence interval, 90% of the confidence intervals will contain the true value of m. H Normally, we make one experiment and try to estimate from this one experiment the confidence interval at a specified CL% Confidence Level…. If in an ensemble of (MC) experiments our estimated Confidence Interval fail to contain the true value of m. H 90% of the cases (for every possible m. H) we claim that our method undercover If in an ensemble of (MC) experiments the true value of m. H is covered within the estimated confidence interval , we claim a coverage 11 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Confidence Interval & Coverage The basic question will come again and again: WHAT IS THE IMPORTANCE OF COVERAGE for a Physicist? The “ problem” : Maybe coverage answers the wrong question… you want to know what is the probability that the Higgs boson exists and is in specific mass range… Excluding a Higgs with a mass<m. H at the 95% CL aims to mean that the interval [0, m. H] does not contain a Higgs Boson with that mass with a 95% probability This is a wrong statement, and the right statement depends on the statistical model. You can never test the signal by itself, only s+b, so your statement is about s(m. H)+b , including the coverage…. We might prefer not to exclude a signal to which we are not sensitive to at the 12 price of undercoverage (CLS) Eilam Gross, Higgs Statistics, Orsay 09

Subjective Bayesian is Good for YOU Thomas Bayes (b 1702) a British mathematician and Presbyterian minister 13 Eilam Gross, Higgs Statistics, Orsay 09

What is the Right Question Is there a Higgs Boson? What do you mean? Given the data , is there a Higgs Boson? Can you really answer that without any a priori knowledge of the Higgs Boson? Change your question: What is your degree of belief in the Higgs Boson given the data… Need a prior degree of belief regarding the Higgs Boson itself… Make sure that when you quote your answer you also quote your prior assumption! Can we assign a probability to a model P(Higgs)? Of course not, we can only assign to it a degree of belief! The most refined question is: Assuming there is a Higgs Boson with some mass m. H, how well the data agrees with that? But even the answer relies on the way you measured the data (i. e. measurement uncertainties), and that might include some pre-assumptions, priors! 14 Eilam Gross, Higgs Statistics, Orsay 09

What is the Right Answer? The Question is: Is there a Higgs Boson? Is there a God? In the book the author uses “divine factors” to estimate the P(Earth|God), a prior for God of 50% He “calculates” a 67% probability for God’s existence given earth… In Scientific American July 2004, playing a bit with the “divine factors” the probability drops to 2%. . . 15 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: The Bayesian Way Can the model have a probability (what is Prob(SM)? ) ? We assign a degree of belief in models parameterized by µ Instead of talking about confidence intervals we talk about 16 credible intervals, where p(µ|x) is the credibility of µ given the data. Eilam Gross, Higgs Statistics, Orsay 09

Systematics Why “download” only music? “Download” original ideas as well…. 17 Eilam Gross, Higgs Statistics, Orsay 09

Systematics is Important An analysis might be killed by systematics 18 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Nuisance Parameters (Systematics) There are two kinds of parameters: Parameters of interest (signal strength… cross section… µ) Nuisance parameters (background cross section, b, signal efficiency) The nuisance parameters carry systematic uncertainties There are two related issues: Classifying and estimating the systematic uncertainties Implementing them in the analysis The physicist must make the difference between cross checks and identifying the sources of the systematic uncertainty. Shifting cuts around and measure the effect on the 19 observable… Very often the observed variation is dominated by the Eilam Gross, Higgs Statistics, Orsay 09 statistical uncertainty in the measurement.

Basic Definitions: Implementation of Nuisance Parameters Implement by marginalizing or profiling Marginalization (Integrating) (The C&H Hybrid) Integrate L over possible values of nuisance parameters (weighted by their prior belief functions -- Gaussian, gamma, others. . . ) Consistent Bayesian interpretation of uncertainty on nuisance parameters Note that in that sense MC “statistical” uncertainties (like background statistical uncertainty) are systematic uncertainties 20 Eilam Gross, Higgs Statistics, Orsay 09

Integrating Out The Nuisance Parameters (Marginalization) Our degree of belief in µ is the sum of our degree of belief in µ given (nuisance parameter), over “all” possible values of That’s a Bayesian way 21 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Priors A prior probability is interpreted as a description of what we believe about a parameter preceding the current experiment Informative Priors: When you have some information about the prior might be informative (Gaussian or Truncated Gaussians…) Most would say that subjective informative priors about the parameters of interest should be avoided (“…. what's wrong with assuming that there is a Higgs in the mass range [115, 140] with equal probability for each mass point? ”) Subjective informative priors about the Nuisance parameters are more difficult to argue with These Priors can come from our assumed model (Pythia, Herwig etc…) These priors can come from subsidiary measurements of the response of the detector to the photon energy, for example. Some priors come from subjective assumptions (theoretical, prejudice symmetries…. ) of our model 22 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Priors – Uninformative Priors: All priors on the parameter of interest are usually uninformative…. Therefore flat uninformative priors are most common in HEP. When taking a uniform prior for the Higgs mass [115, ]… is it really uninformative? do uninformative priors exist? When constructing an uninformative prior you actually put some information in it… But a prior flat in the coupling g will not be flat in ~g 2 Depends on the metric! Note, flat priors are improper and might lead to serious problems of undercoverage (when one deals with >1 channel, i. e. beyond counting, one should AVOID them) 23 Eilam Gross, Higgs Statistics, Orsay 09

DISCOVERY 24 Eilam Gross, Higgs Statistics, Orsay 09

Basic Definitions: Rayleigh Distribution 25 Eilam Gross, Higgs Statistics, Orsay 09

The Discovery Case Study • We assume a Gaussian “Higgs” signal (s) on top of a Rayleigh shaped background (b) NOTE: b=b( ) • The signal strength is µ µ=1, SM Higgs • µ=0, SM without Higgs • • Two hypothetical measurements • Data m. H 26 Eilam Gross, Higgs Statistics, Orsay 09

The Discovery Case Study • BG control sample scaled to the expected BG via a factor NOTE: b=b( ) • The control sample constraints the background m. H 27 Eilam Gross, Higgs Statistics, Orsay 09

A Simultaneous Fit Two measurements m. H 28 Eilam Gross, Higgs Statistics, Orsay 09 m. H

m. H 29 Eilam Gross, Higgs Statistics, Orsay 09 m. H

The Discovery Case Study Note, in this example, the signal towards the end of the background mass distribution (m. H=20, 80) is better separated from the signal near the middle (m. H=50). m. H 30 m. H Eilam Gross, Higgs Statistics, Orsay 09 m. H

LR motivation. The Neyman-Pearson Lemma When performing a hypothesis test between two simple hypotheses, H 0 and H 1, the Likelihood Ratio test, which rejects H 0 in favor of H 1, is the most powerful test of size for a threshold Define a test statistic Note: Likelihoods are functions of the data, even though we often not specify it explicitly 31 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist LR (CLb) method • Define a test statistics 32 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist LR (CL) method • Define a test statistics • Use MC to generate the pdf of under H 0(B only) and H 1 (S+B) e lu a -v p 33 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist LR (CL) method • Define a test statistics • Use MC to generate the pdf of under H 0(B only) and H 1 (S+B) • Let obs be a result of one experiment (LHC) e lu a -v p log observed 34 Eilam Gross, Higgs Statistics, Orsay 09

Example (from SUSY 02): Simulating BG Only Experiments The likelihood ratio, -2 ln (m. H) tells us how much the outcome of an experiment is signal-like NOTE, here the s+b pdf is plotted to the left! Discriminator (from SUSY 02): -2 ln (m. H) 35 Eilam Gross, Higgs Statistics, Orsay 09 s+b like b-like

Example (from SUSY 02): Simulating S(m. H)+b Experiments The likelihood ratio, -2 ln (m. H) tells us how much the outcome of an experiment is signal-like NOTE, here the s+b pdf is plotted to the left! 36 s+b like Eilam Gross, Higgs Statistics, Orsay 09 -2 ln (m. H) b-like

Straighten Things Up s+b like 37 b-like Eilam Gross, Higgs Statistics, Orsay 09 s+b like

The frequentist LR (CL) method • Define a test statistics • Use MC to generate the pdf of under • • H 0(B only) and H 1 (S+B) Let be a result of one experiment (LHC) The p-value is the probability to get an observation which is less b-like than the observed one If the result of the experiment (LHC) yields a p-value< 2. 8· 10 -7 a 5 discovery is claimed NOTE: the p-value can be interpreted as a frequency this is a frequentist approach 38 Eilam Gross, Higgs Statistics, Orsay 09 e lu a -v p log observed

Profile Likelihood Let us consider a counting experiment, measuring n Define Construct a Likelihood Test the bg-only ( µ=0 ) hypothesis 39 Eilam Gross, Higgs Statistics, Orsay 09

Profile Likelihood Define the PL ratio Define the test statistics to be Note: If the data is b-like If the data contains a signal b-like 40 q 0 s+b-like Eilam Gross, Higgs Statistics, Orsay 09

The Profile Likelihood Simulator The Profile Likelihood Matlab Simulator © O. Vitells & E. Gross 41 Eilam Gross, Higgs Statistics, Orsay 09

Systematics Normally, the background, b( ), has an uncertainty which has to be taken into account. In this case is called a nuisance parameter (which we associate with background systematics) How can we take into account the nuisance parameters? One way: marginalize them (integrate them out using priors) the Hybrid CL (mix frequentist and Bayesian approach) prior Another way is profiling via the MLEs: 42 Eilam Gross, Higgs Statistics, Orsay 09

Systematics via Profiling is the MLE of under H 1 under H 0 PL Ratio LR (CLs+b) 43 Eilam Gross, Higgs Statistics, Orsay 09

Wilks Theorem Under a set of regularity conditions and for a sufficiently large data sample, Wilks’ theorem says that for a hypothesized value of μ, the pdf of the statistic − 2 lnλ (μ) approaches the chi-square pdf for one degree of freedom One of the conditions is Hµ H 0 44 Eilam Gross, Higgs Statistics, Orsay 09

Wilks Theorem distributes as a 2 with 1 d. o. f for experiments under the hypothesis Hµ 2 for b-only experiments, q 1 distributes as 2 for s+b experiments i. e. q 0 distributes as This ensures simplicity, coverage, speed 45 Eilam Gross, Higgs Statistics, Orsay 09

Wilks Theorem distributes as a 2 with 1 d. o. f for experiments under the hypothesis Hµ Z is the significance 46 Eilam Gross, Higgs Statistics, Orsay 09

Discovery - Illustrated 47 Eilam Gross, Higgs between Statistics, the Orsay 09 and the no-Higgs hypothesis p 0 is the level of compatibility data If p 0 is smaller than ~2. 8∙ 10 -7 we claim a 5 discovery

Median Sensitivity To estimate the median sensitivity of an experiment (before looking at the data), one can either perform lots of s+b experiments and estimate the median q 0, med or evaluate q 0 with respect to a representative data set, the ASIMOV data set with m=1, i. e. n=s+b 48 Eilam Gross, Higgs Statistics, Orsay 09

The Profile Likelihood Simulator The Profile Likelihood Matlab Simulator © O. Vitells & E. Gross 49 Eilam Gross, Higgs Statistics, Orsay 09

The ASIMOV data sets The name of the Asimov data set is inspired by the short story Franchise, by Isaac Asimov [1]. In it, elections are held by selecting a single voter to represent the entire electorate. The "Asimov" Representative Data-set for Estimating Median Sensitivities with the Profile Likelihood G/ Cowan, K. Cranmer, E. Gross , O. Vitells, under preparation [1] Isaac Asimov, Franchise, in Isaac Asimov: The Complete Stories, Vol. 1, Broadway Books, 1990. 50 Eilam Gross, Higgs Statistics, Orsay 09

The Profile Likelihood Simulator The Profile Likelihood Matlab Simulator © O. Vitells & E. Gross 51 Eilam Gross, Higgs Statistics, Orsay 09

Wilks Theorem with Nuisance Parameters if there are n parameters of interest, i. e. , those parameters that do not get a double hat in the numerator of the likelihood ratio then − 2 lnλ (μ) asymptotically follows a chi-square distribution for n degrees of freedom. 52 Eilam Gross, Higgs Statistics, Orsay 09

The Profile Likelihood with Nuisance Parameters The procedure is identical to the one without nuisance parameters 53 Eilam Gross, Higgs Statistics, Orsay 09

CL & CI - Wikipedia A confidence interval (CI) is a particular kind of interval estimate of a population parameter. Instead of estimating the parameter by a single value, an interval likely to include the parameter is given. Thus, confidence intervals are used to indicate the reliability of an estimate. How likely the interval is to contain the parameter is determined by the confidence level or confidence coefficient. Increasing the desired confidence level will widen the confidence interval. 54 Eilam Gross, Higgs Statistics, Orsay 09

The Profiled CL way Generate the pdf of -2 ln PL under H 0 and H 1 Define CI for q 0: [- , qobs] The CL of this CI is given by p 0 ''='' 1 -CLb ATLAS, CERN – Open 2008 -029 Cowan, Cranmer, E. G. , Vitells, in preparation 55 Eilam Gross, Higgs Statistics, Orsay 09

The Profiled CL way p 0 ''='' 1 -CLb • The median significance can be obtained with the one Asimov data set n~s+b, m~b ATLAS, CERN – Open 2008 -029 Cowan, Cranmer, E. G. , Vitells, in preparation 56 Eilam Gross, Higgs Statistics, Orsay 09

The Profiled CL way In this example a Higgs with a mass m. H<32 or m. H>52 is expected to be discovered, i. e. if the Higgs exists in this mass range it will be discovered >50% of hypothetical LHC experiments 57 Eilam Gross, Higgs Statistics, Orsay 09

The Profile Likelihood Ratio • CL Profiled LR PROFILE LIKELIHOOD RATIO IS DIFFERENT! • PL Ratio: • Test the null H 0 hypothesis • Use the Wilks theorem and the Asimov data set to deduce the p 0, med 58 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist Profile Likelihood Ratio vs Profiled CL • Its not surprising that using a modified LR (not the Neyman-Pearson motivated LR) gives a slightly less sensitive result , yet • Why using a method with a slightly lower sensitivity? • Because of the Wilks theorem which can save us hours and days of computer time 59 Eilam Gross, Higgs Statistics, Orsay 09 Profiled CL PL Ratio

Expected Discovery Sensitivity Combination of Channels Each channel has its PLR with respect to the Asimov data set (x=s+b) For each channel We find That is, l. A(0|x=s+b) approximates the median value of l (0) one would obtain from data generated according to the s+b hypothesis. The value of l. A(0|x=s+b) is used to determine the median q 0, med, which is used to find the median p-value, pmed. 60 Eilam Gross, Higgs Statistics, Orsay 09

The Look Elsewhere Effect 61 Eilam Gross, Higgs Statistics, Orsay 09

Look Elsewhere Effect • To establish a discovery we try to reject the background only hypothesis H 0 against the alternate hypothesis H 1 • H 1 could be • A Higgs Boson with a specified mass m. H • A Higgs Boson at some mass m. H in the search mass range • The look elsewhere effect deals with the floating mass case Let the Higgs mass, m. H, and the signal strength µ be 2 parameters of interest 62 Eilam Gross, Higgs Statistics, Orsay 09

Look Elsewhere Effect 2 parameters of interest: the signal strength µ and the Higgs mass m. H 63 m. H Eilam Gross, Higgs Statistics, Orsay 09

Look Elsewhere Effect • Letting the Higgs mass float we find that the background-only experiments distribute approximately as a • The median sensitivity is given by the corresponding p-value • • Does it satisfy Wilks theorem? 64 Eilam Gross, Higgs Statistics, Orsay 09

Look Elsewhere Effect Back of the envelope: m. H 65 Eilam Gross, Higgs Statistics, Orsay 09

Discovery via the Bayes Way: Bayes Factors 66 Eilam Gross, Higgs Statistics, Orsay 2009

The Bayes Way 67 Eilam Gross, Higgs Statistics, Orsay 09

Frequentist ~ Bayesian ? 68 Eilam Gross, Higgs Statistics, Orsay 09

Frequentist ~ Bayesian ? We found With some work (E. G. , O. Vitells, POS Krakow 09) we show that for the Asimov data set (s+b) Finally we find 69 Eilam Gross, Higgs Statistics, Orsay 09

Frequentist ~ Bayesian ? 70 Eilam Gross, Higgs Statistics, Orsay 09

Frequentist~Bayesian Why Does It Work This relationship between the Bayes factor and the frequentist PL ratio, though disturbing in the first place, is not surprising when you come to think about it. Wilks theorem ensures that using the PL ratio you do not need to perform any toy MC experiments to tell a significance of an observation based on the one observed data set. This is also the characteristics of a Bayesian hypothesis test. 71 Eilam Gross, Higgs Statistics, Orsay 09

EXCLUSION 72 Eilam Gross, Higgs Statistics, Orsay 09

Exclusion Case Study M=20 m. H 73 M=50 M=80 m. H Eilam Gross, Higgs Statistics, Orsay 09 m. H

Exclusion and p-value To exclude the s(m. H)+b hypothesis (H 1) we try to reject it i. e. calculating the observed or expected p-value of the test statistic under the s(m. H)+b pdf If the p-value ps+b<5% we claim the s+b hypothesis was rejected at >95% CL Here we use the arguable relationship CL=1 -ps+b 74 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist CLs+b method Generate the pdf of -2 ln under H 0 and H 1 Test the H 1 hypothesis Define CI for q 1: [- , qobs] The CL of this CI is given by p 1 ''='' CLs+b 75 Eilam Gross, Higgs Statistics, Orsay 09

The frequentist CLs+b method Use the LR as a test statistics To take systematics unto account integrate the nuisance parameters or profile them The exclusion is given by the e Eilam Gross, Higgs Statistics, Orsay 09 lu 76 va p- s(m. H)+b hypothesis p-value ps+b=CLs+b If ps+b<5%, the s(m. H)+b hypothesis is rejected at the 95% CL p-value=CL s+b

Exclusion – Illustrated 77 Eilam Higgs Statistics, 09 and the Higgs hypothesis p 1 is the level of Gross, compatibility between. Orsay the data If p 1 is smaller than 0. 05 we claim an exclusion at the 95% CL

Exclusion with Profile Likelihood Exclusion is related to the probability of the “would be” signal to fluctuate down to the background only region (i. e. the p-value of the s+b hypothesis) To evaluate the median sensitivity of an experiment we generate a BG only data and calculate the median q 1, med p 1, med Zmed. Exclusion at the 95% C. L. (one sided) means Z=1. 64 78 Eilam Gross, Higgs Statistics, Orsay 09

Deriving an Upper Limit µ is the signal strength When measured it can be interpreted as To get a “ 95% CL” upper limit on µ one has to solve This will give the 95% credible (or confidence) interval [0, µ 95] If this interval contains the value µ=1, the SM Higgs is NOT excluded This µ 95 is therefore interpreted as a 95% upper limit on µ, i. e. µupper. 79 Eilam Gross, Higgs Statistics, Orsay 09

Exclusion Expected Limit, Combination of Channels Each channel has its PLR with respect to the Asimov data set (background only, x=b) For each channel We find That is, l. A(m|x=b) approximates the median value of l (m) one would obtain from data generated according to the background-only hypothesis. The value of l. A(m|x=b) is used to determine the median qmed, which is used to find the median p-value, pmed. This has to be computed for all µ and the point where pµmed = 0. 05 gives the 95% CL upper limit on µmed. 80 Eilam Gross, Higgs Statistics, Orsay 09

Profile Likelihood Ratio Test the S(m. H)+b hypothesis i. e. test the µ=1 hypotheis • q 1 distributes as a 2 under s(m. H)+b experiments (H 1) • The exclusion significance can be expressed in terms of an equivalent exclusion CL • The exclusion sensitivity is 81 the If ps+b<5% we (using a wrong jargon) say that the signal is excluded at >95% CL (CL=1 -ps+b) Note that the exclusion CL is identified here with CL=1 -CLs+b where CLs+b=ps+b median CL, and using Eilam Gross, Higgs Statistics, Orsay 09 toy MCs one can find the 1

Exclusion Profile Likelihood Ratio • A Higgs with a specific mass m. H is excluded at the 95% CL if the observed p-value of the s(m. H)+b hypothesis is below 0. 05 If ps+b<5%, the s(m. H)+b hypothesis is rejected at the 95% CL • In this example a Higgs Boson is expected to be excluded p 1<0. 05 (CL>95%) in all the mass range 82 Eilam Gross, Higgs Statistics, Orsay 09

Exclusion Bayesian Let be the posterior for µ NOTE: The pdf of the posterior is based on the one observed data event with the likelihood integrated over the nuisance parameters To set an upper limit on the signal strength calculate the credibility interval [0, µ 95] 83 Eilam Gross, Higgs Statistics, Orsay 09 Data = Asimov b

Exclusion Bayesian Let be the posterior for µ NOTE: The toy MC are needed just to find the median sensitivity, but once the data is delivered, it is sufficient to determine the upper limit using the posterior integration 84 Eilam Gross, Higgs Statistics, Orsay 09 Data = b-only

Exclusion Bayesian • We find that the credibility interval [0, µ 95] does not contain µ 95=1 (SM) for m. H<28 or m. H>61 • This is sometimes wrongly expressed as an exclusion at the 95% CL m. H 85 Eilam Gross, Higgs Statistics, Orsay 09

Comparing Bayesian to Frequentist PL saddle-point approximation (for flat priors) For the Asimov BG only data Note: Taking the proper normalizations into account we find the following equivalence: 86 Eilam Gross, Higgs Statistics, Orsay 09

Exclusion Bayesian vs PL Ratio • Comparing a credibility Bayesian interval to 95% frequentist CL is like comparing oranges to apples…. Yet • In Bayesian statistics, the observed data is sufficient to infer how strong is an hypothesis (assuming some priors). • In the Profile Likelihood frequentist approach, Wilks’ theorem ensures that under a set of regularity conditions and for a sufficiently large data sample, for m. H an hypothesized value of µ, the • NOTE: One has to be careful about the 1 sided vs 2 -sided significance 87 Eilam Gross, Higgs Statistics, Orsay 09

The problem with the CLs+b method Use the LR as a test statistics To take systematics unto account integrate the nuisance parameters or profile them The exclusion is given by the va pe lu s(m. H)+b hypothesis p-value ps+b=CLs+b If ps+b<5%, the s(m. H)+b hypothesis is rejected at the 95% CL p-value=CLs+b 88 Eilam Gross, Higgs Statistics, Orsay 09

A wrong jargon – mixing p-values and Confidence Levels b-like 90 Eilam Gross, Higgs Statistics, Orsay 09 s+b like

CLs+b and CLb 1 -CLb is the p value of the b- hypothesis, i. e. the probability to get a result less compatible with the BG only hypothesis than the observed one (in experiments where BG only hypothesis is true) CLs+b is the p-value of the s+b hypothesis, i. e. the probability to get a result which is less compatible with a Higgs signal when the signal hypothesis is true! A small CLs+b leads to an exclusion of the signal hypothesis at the CL=1 - CLs+b confidence level. 91 b-like Eilam Gross, Higgs Statistics, Orsay 09 s+b like Observed Likelihood

The Problem of Small Signal <Nobs>=s+b leads to the physical requirement that Nobs>b A very small expected s might lead to an anomaly when Nobs fluctuates far below the expected background, b. At one point DELPHI alone had CLs+b=0. 03 for m. H=116 Ge. V However, the cross section for 116 Ge. V Higgs at LEP was too small and Delphi actually had no sensitivity to observe it The frequntist would say: Suppose there is a 116 Ge. V Higgs…. Only 3% of the confidence intervals contain the true value of q. SM(m. H) i. e. in 3% of the experiments the true signal would be rejected… (one would obtain a result incompatible or more Observed Likelihood so with m=116) i. e. a 116 Ge. V Higgs is excluded at the 97% CL…. . 92 97% of the intervals [qobs, ] do not contain q. SM (m. H) Eilam Gross, Higgs Statistics, Orsay 09

The CLs Method for Upper Limits Inspired by Zech(Roe and Woodroofe)’s derivation for counting experiments A. Read suggested the CLs method with In the DELPHI example, CLs=0. 03/0. 13=0. 26, i. e. a 116 Ge. V could not be excluded at the 97% CL anymore…. . (pb=1 -CLb=0. 87) Observed Likelihood 93 Eilam Gross, Higgs Statistics, Orsay 09

The Meaning of CLs False exclusion rate of Signal when Signal is true Is it really that bad that a method undercovers where Physics is sort of handicapped… (due to loss of sensitivity)? 94 m. H Eilam Gross, Higgs Statistics, Orsay 09 115 Ge. V

The modified frequentist CLs In this example, while using PL or the CLs the Higgs is excluded in all the mass range, the CLs reduces the sensitivity and does not allow to exclude a Higgs with 30<m. H<60 • 95 Eilam Gross, Higgs Statistics, Orsay 09

Conclusions We have explored and compared all the methods to test hypotheses that are currently in use in the High Energy Physics market (PLR, CLs+b, CLs, Bayesian ) We have shown that all methods tend to give similar results, (for both exclusion and discovery using flat priors) weather one integrates the nuisance parameters or profile them Even though we have used typical case studies, real life might be different and all available methods should be explored 96 Eilam Gross, Higgs Statistics, Orsay 09