Everything You Always Wanted To Know About Limits

Everything You Always Wanted To Know About Limits* Roger Barlow Manchester University YETI 06 *But were afraid to ask Everything you wanted to know Roger Barlow: YETI 06 about limits

Summary Prediction confronts data & sees small/zero signal Bayesian Probability (Health Warning) Frequentist probability and Confidence Level language Zero events Few events: Confidence belt Roger Barlow: YETI 06 Likelihood Gaussian ln L= -½ Extension to several parameters The horrendous case of large backgrounds Everything you wanted to know about limits 2

Model predictions Input model and parameters Low energy Lagrangian Cuts designed to bring out signal Feynman Rules for Feynman diagrams Cross Sections and Branching Ratios Monte Carlo programs Experiment duration, luminosity, Efficiency etc Number of events Roger Barlow: YETI 06 Everything you wanted to know about limits Data 3

What happens if there’s nothing there? Even if your analysis finds no events, this is still useful information about the way the universe is built Want to say more than: “We looked for X, we didn’t see it. ” Need statistics – which can’t prove anything. “We show that X probably has a mass greater than. . /a coupling smaller than…” Roger Barlow: YETI 06 Everything you wanted to know about limits 4

Probability(1): Frequentist Define Probability of X as P(X)=Limit N ∞ N(X)/N Examples: coins, dice, cards For continuous x extend to Probability Density P(x to x+dx)=p(x)dx Examples: • Measuring continuous quantities (p(x) often Gaussian) • Parton momentum fractions (proton pdfs) Roger Barlow: YETI 06 Everything you wanted to know about limits 5

Digression: likelihood Probability distribution of random variable x often depends on some parameter a. Joint function p(x, a) Considered as p(x)|a this is the pdf. Normalised: ∫p(x)dx=1 Considered as p(a)|x this is the Likelihood L(a) Not ‘likelihood of a’ but ‘likelihood that a would give x’ Not normalised. Indeed, must never be integrated. Roger Barlow: YETI 06 Everything you wanted to know about limits 6

Limitation of Frequentist Probability Can’t say “It will probably rain tomorrow. ” There is only one tomorrow. P is either 1 or 0 Have to say “The statement ‘It will rain tomorrow. ’ is probably true. ” Can then even quantify (meteorology). Roger Barlow: YETI 06 Everything you wanted to know about limits 7

Interpreting physics results: Mt =173± 2 Ge. V/c 2 Can’t say ‘Mt has a 68% probability of lying between 171 and 175 Ge. V/c 2’ Have to say ‘The statement “Mt lies between 171 and 175 Ge. V/c 2”has a 68% probability of being true’ i. e. if you always say a value lies within its error bars, you will be right 68% of the time. Say “Mt lies between 171 and 175 Ge. V/c 2” with 68% Confidence. Or 169 -177 with 95% confidence. Or… Roger Barlow: YETI 06 Everything you wanted to know about limits 8

Interpreting null result Your analysis searches for events. Sees none. Use Poisson formula: P(n; )=e- n/n! Small could well give 0 events • =0. 5 gives P(0)=61% • =1. 0 gives P(0)=37% • =2. 3 gives P(0)=10% • =3. 0 gives P(0)=5% If you always say ‘ 3. 0’ you will be right (at least) 95% of the time. 3. 0 – with 95% confidence (a. k. a 5% significance. ) ‘If is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less. ’ given by model parameters. Limit on translates to limit on mass, coupling, , branching ratio or whatever 9 Roger Barlow: YETI 06 Everything you wanted to know about limits

Probability(2): Bayesian P(X) expresses by degree of belief in X Can calibrate against cards, dice, etc. Extend to probability density p(x) as before No restrictions on X or x. Rain, MT, MH, whatever Interpret physics results using Bayes’ Theorem: pposterior(a|data) p(data|a) x pprior(a) Roger Barlow: YETI 06 Everything you wanted to know about limits 10

Bayes at work Zero events seen x = Posterior P( ) P(0 events| ) Prior: uniform (Likelihood) ò 3 0 Roger Barlow: YETI 06 P( ) d = 0. 95 Same as Frequentist limit Happy coincidence Everything you wanted to know about limits 11

Bayes at work again Is that uniform prior really credible? x = P(0 events| ) Posterior P( ) Prior: uniform in ln Upper limit totally different! ò 3 0 Roger Barlow: YETI 06 P( ) d >> 0. 95 Everything you wanted to know about limits 12

Bayes: the bad news • The prior affects the posterior. It is your choice. That makes the measurement subjective. This is BAD. (We’re physicists, dammit!) • A Uniform Prior does not get you out of this. • SUSY ‘parameter space’ is not a ‘phase space’ • Attempts to invent universally-agreed priors (‘Objective’ and/or ‘Reference’ Priors) have not worked Better news: If there is a lot of data then the prejudicial effects of the choice of prior can be small. • This should ALWAYS be checked for (‘robustness under choice of prior’. ) Roger Barlow: YETI 06 Everything you wanted to know about limits 13

Frequentist versus Bayesian? Statisticians do a lot of work with Bayesian statistics and there a lot of useful ideas. But they are careful about checking for robustness under choice of prior. Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness. Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness. ) But ALWAYS be aware which you are using. Roger Barlow: YETI 06 Everything you wanted to know about limits 14

A Gaussian Measurement No problems p(x)=exp[-(x- )2/2 2]/ √ 2 x: symmetric x is within ± of with 68% probability is within ± of x at 68% confidence x is above -1. 645 with 95% probability is below x+1. 645 at 95% confidence Choice of confidence level and arrangement Can read regions off log likelihood plot as L(a)=exp[-(x- )2/2 2]/ √ 2 Ln L -( -x)2/2 2 68% region corresponds to fall of ½ from peak Roger Barlow: YETI 06 Everything you wanted to know about limits 15

A Poisson measurement You detect 5 events. Best value 5. But what about the errors? 1. 5±√ 5=5± 2. 24 Assumes e - n/n! is Gaussian in n. True only for large - and 5 is small 2. Find points where log likelihood falls by ½. Assumes e- n/n! is Gaussian in . Gives upper error of 2. 58, lower error of 1. 92 Roger Barlow: YETI 06 Everything you wanted to know about limits 16

3: Doing it properly: Confidence belt (Neyman interval) + Use e- n/n! 68% 16% For any true the 16% probability that (n, ) is within the belt is 68% (or more) by construction For any n, lies in [ -, +] at 68% confidence n Technique works for Get upper error 3. 38, lower any CL, and single or error 2. 16 double sided - Roger Barlow: YETI 06 Everything you wanted to know about limits 17

Consumer guide ln L =- ½ is a standard and easy to use. Fine for everyday use. (Though for a simple count the Neyman limit is quite easy) For 90% 1 -sided (upper) limit use ln L =-0. 82 (1. 28 ) For 95% use ln L =-1. 35 (1. 645 ) Just plot the likelihood and read off the value. Then translate back to model parameters Roger Barlow: YETI 06 Everything you wanted to know about limits 18

Frequency method: the big problem Observe 5 events. Expected background of 0. 9 events. Data = signal + background Say with 68% confidence: data in range 2. 84 to 8. 38 So say with 68% confidence: signal in range 1. 94 to 7. 48 Suppose expected* background 4. 9. Or 6. 9. Or 10. 9 ? “We say at 68% confidence that the number of signal events lies between -8. 06 and -2. 52” This is technically correct. We are allowed to be wrong 32% of the time. But stupid. We know that the background happens to have a downward fluctuation but have no way of incorporating that knowledge *We assume that the background is calculated correctly Roger Barlow: YETI 06 Everything you wanted to know about limits 19

Strategy 1: Bayes Prior is uniform for positive , zero for negative . No problem. Get requirement (for n observed, known background b, 90% upper limit) 0. 1= nexp(- +-b) ( ++b)r/r! nexp(-b) br/r! Known as “the old PDG formula” or “Helene’s formula” or “that heap of crap” Roger Barlow: YETI 06 Everything you wanted to know about limits 20

Strategy 2: Feldman-Cousins Also called* ‘the Unified Approach’ Real physicists wait to see their result and then decide whether to quote an upper limit or a range. This ‘flip-flopping’ invalidates the method. They provide a procedure that incorporates it automatically, and always gives non-stupid results. Critics say (1) can lead to experiments quoting a range when they’re not claiming a discovery (2) is computationally intensive and (3) For zero observed events, the higher the background estimate the better (i. e. lower) the limit on signal * By Feldman and Cousins, principally Roger Barlow: YETI 06 Everything you wanted to know about limits 21

Strategy 3: CLs As used by LEP Higgs working group Generalisation of Helene formula Some quantity Q. Could be number of events, or something more clever CLb=P(Q or less|b) CLs+b=P(Q or less|s+b) CLs=CLs+b/CLb Used as confidence level. Optimise strategy using it and quote results Roger Barlow: YETI 06 Everything you wanted to know about limits 22

2(+) parameters Fix b, find 68% confidenceb range for a, using ln L=-½ Fix a, find 68% range for b Combination (square) has 0. 682=46% L(a, b) a ln L=-½ circle has 39% Confidence Define regions through contours of log L – Confidence content given by 2 ln L= 2 for which P( 2 , n)=CL Caution! Cannot redefine a as b+c+d, claim 3 parameters and cut with P( 2, 3) instead of P( 2 , 1) Roger Barlow: YETI 06 Everything you wanted to know about limits 23

Summary Prediction confronts data & sees small/zero signal Bayesian Probability (Health Warning) Frequentist probability and Confidence Level language Zero events Few events: Confidence belt Roger Barlow: YETI 06 Likelihood Gaussian ln L= -½ Extension to several parameters The horrendous case of large backgrounds Everything you wanted to know about limits 24

Remember! Zero events = 95% CL upper limit of 3 events If it’s more involved, plot the likelihood function and use ln L=-½ for 68% central, etc Be suspicious of anything you don’t understand If you’re integrating the likelihood you are a Bayesian. I hope you know what you’re doing. Roger Barlow: YETI 06 Everything you wanted to know about limits 25

Further Reading • Workshop on Confidence Limits, CERN yellow report 2000 -005 • Proc. Conf. Advanced Statistical Techniques in Particle Physics, Durham, IPPP 02/39 • Proc. PHYSTAT 03 – SLAC-R-703 • Proc PHYSTAT 05, Oxford forthcoming Roger Barlow: YETI 06 Everything you wanted to know about limits 26