Multivariate Analysis with TMVA Basic Concept and New

Multivariate Analysis with TMVA Basic Concept and New Features Helge Voss(*) (MPI–K, Heidelberg) LHCb MVA Workshop, CERN, July 10, 2009 (*) On behalf of the author team: A. Hoecker, P. Speckmayer, J. Stelzer, J. Therhaag, E. v. Toerne , H. Voss And the contributors: See acknowledgments on page xx On the web: http: //tmva. sf. net/ (home), https: //twiki. cern. ch/twiki/bin/view/TMVA/Web. Home (tutorial)

Event Classification § Suppose data sample of two types of events: with class labels Signal and Background (will restrict here to two class cases. Many classifiers can in principle be extended to several classes, otherwise, analyses can be staged) § how to set the decision boundary to select events of type S ? § we have discriminating variables x 1, x 2, … Rectangular cuts? x 2 A linear boundary? x 2 S B A nonlinear one? x 2 S B x 1 § How can we decide? § Once decided on a class of boundaries, how to find the “optimal” one? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA x 1 2

Regression In High Energy Physics § Most HEP event reconstruction require some sort of “function estimate”, of which we do not have an analytical expression: § § § energy deposit in a the calorimeter: shower profile calibration entry location of the particle in the calorimeter distance between two overlapping photons in the calorimeter … you can certainly think of more. . Maybe you could even imagine some final physics parameter that needs to be fitted to your event data and you would rather use a non analytic fit function from Monte Carlo events than an analytic parametrisation of theory ? ? ? : § § reweighing method used in the LEP W-mass analysis … somewhat wild guessing, might not be reasonable to do. . Maybe your application can be successfully learned by a machine using Monte Carlo events ? ? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 3

Regression § how to estimate a “functional behaviour” from a given set of ‘known measurements” ? § Assume for example “D”-variables that somehow characterize the shower in your calorimeter. § from Monte Carlo or testbeam measurements, you have a data sample with events that measure all D-Variables x and the corresponding particle energy § the particle energy as a function of the D observables x is a surface in the D+1 dimensional space given by D-observables + energy f(x) constant ? § seems trivial? ? linear? f(x) non - linear? x x x § maybe… the human eye and brain behind have very good pattern recognition capabilities! § but what if you have more variables? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 4

Event Classification: Why MVA ? • Assume 4 Uncorrelated Gaussians: • What do you think. Is this event is more “likely” signal or background ? ? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 5

Event Classification § Each event, if Signal or Background, has “D” measured variables. D “feature space” § Find a mapping input/observable/”feature” space y(x): Rn R: from D-dimensional most general form to one dimensional output class lables y = y(x); x D x={x 1, …. , x. D}: input variables § If one histogramms the resulting y(x) values: § Who sees how this would look like for the regression problem? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA y(x) 6

Event Classification § Each event, if Signal or Background, has “D” measured variables. § Find a mapping from D-dimensional input/observable/”feature” space to one dimensional output y(B) 0, y(S) 1 class lables y(x): Rn R: D “feature space” § y(x): y(x) “test statistic” in D-dimensional space of input variables § distributions of y(x): PDFS(y) and PDFB(y) § used to set the selection cut! efficiency and purity y(x): >cut: signal =cut: decision boundary <cut: background § y(x)=const: surface defining the decision boundary. § overlap of PDFS(y) and PDFB(y) separation power , purity LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 7

MVA and Machine Learning The previous slide was basically the idea of “Multivariate Analysis” (MVA) rem: What about “standard cuts” ? Finding y(x) : Rn R ( “basically” the same for regression and classification) given a certain type of model class y(x) in an automatic way using “known” or “previously solved” events i. e. learn from known “patterns” such that the resulting y(x) has good generalization properties when applied to “unknown” events that is what the “machine” is supposed to be doing: supervised machine learning Of course… there’s no magic, we still need to: choose the discriminating variables choose the class of models (linear, non-linear, flexible or less flexible) tune the “learning parameters” bias vs. variance trade off check generalization properties consider trade off between statistical and systematic uncertainties LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 8

Overview/Outline Idea: rather than just implementing new MVA techniques and making them available in ROOT (i. e. like TMulit. Layer. Percetron does): one common platform / interface for all MVA classification and regression algorithms. easy to use and switch between classifiers (common) data pre-processing capabilities train / test all classifiers on same data sample and evaluate consistently common analysis, application and evaluation framework (ROOT Macros, C++ executables, python scripts. . ) application of trained classifier/regressor using lib. TMVA. so and corresponding “weight files” or stand alone C++ code (latter is currently disabled (TMVA 4. 01) ) Outline The TMVA project Survey of the available classifiers/regressors Toy examples General considerations and systematics LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 9

TMVA Development and Distribution TMVA is a sourceforge (SF) package for world-wide access Home page ………………. http: //tmva. sf. net/ SF project page …………. http: //sf. net/projects/tmva View SVN …………………http: //tmva. svn. sourceforge. net/viewvc/tmva/trunc/TMVA/ Mailing list. ………………. . http: //sf. net/mail/? group_id=152074 Tutorial TWiki ……………. https: //twiki. cern. ch/twiki/bin/view/TMVA/Web. Home Active project Currently 6 core developers, and 42 registered contributors at SF >4864 downloads since March 2006 (without CVS/SVN checkouts and ROOT users) current version. 4. 0. 1 (29 Jun 2009) try to be fast on users requests bug fixes etc. Integrated and distributed with ROOT since ROOT v 5. 11/03 (newest version 4. 0. 1 in ROOT production release 5. 23) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 10

New (and future) TMVA Developments MV-Regression Code re-design allowing for: boosting any classifier (not only decision trees) combination of any pre-processing algorithms classifier training in multi-regions multi-classifiers (will slowly creep in to some classifiers…) cross-validation New MVA-Technique PDE-Foam (classification + regression) Data preprocessing Gauss-transformation arbitrary combination of preprocessing Numerous small add-ons etc. . LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 11

The TMVA Classifiers (Regressors) Currently implemented classifiers : Rectangular cut optimisation Projective (1 D) likelihood estimator multidimensional likelihood estimators (PDE range search, PDE-Foam, k-Nearest Neighbour) (also as Regressor) Linar discirminant analysis (Fisher, H-Matrix general Linear (also as Regressor) ) Function Discriminant (also as Regressor) Artificial neural networks (3 multilayer perceptron implementations (MLP also as Regressor with multi-dimensional target) ) Boosted/bagged/randomised decision trees (also as Regressor) Rule Ensemble Fitting Support Vector Machine (Plugin for any other MVA classifier (e. g. Neuro. Bayes)) Currently implemented data preprocessing : Linear decorrelation Principal component analysis Gauss transformation LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 12

Example for Data Preprocessing: Decorrelation Commonly realised for all classifiers in TMVA (centrally in Data. Set class) Removal of linear correlations by rotating input variables using the “square-root” of the correlation matrix using the Principal Component Analysis Note that decorrelation is only complete, if Correlations are linear Input variables are Gaussian distributed original LHCb MVA Workshop, CERN, July 10, 2009 SQRT derorr. H. Voss ― Multivariate Analysis with TMVA PCA derorr. 13

“Gaussian-isation” § Improve decorrelation by pre-Gaussianisation of variables First: transformation to achieve uniform (flat) distribution: Transformed variable k Measured value PDF of variable k The integral can be solved in an unbinned way by event counting, or by creating non-parametric PDFs (see later for likelihood section) Second: make Gaussian via inverse error function: Third: decorrelate (and “iterate” this procedure) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 14

“Gaussian-isation” Background Signal -Original Gaussianised - Gaussianised We cannot simultaneously Gaussianise both signal and background ? ! LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 15

Pre-processing The decorrelation/gauss transform can be determined from either: Signal + Background together Signal only Background only For all but the “Likelihood” methods (1 D or multi-dimensional) we need to decide on either of those. (TMVA default: signal+background) For “Likelihood” methods where we test “Signal” hypothesis against “Background” hypothesis, BOTH are used. LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 16

Rectangular Cut Optimisation Simplest method: indiviudual cuts on each variable cuts out a single rectangular volume from the variable space Technical challenge: how to find optimal cuts ? MINUIT fails due to non-unique solution space TMVA uses: Monte Carlo sampling, Genetic Algorithm, Simulated Annealing Huge speed improvement of volume search by sorting events in binary tree Cuts usually benefit from prior decorrelation of cut variables LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 17

Projective (1 D) Likelihood Estimator (PDE Approach) Probability density estimators for each input variable combined in likelihood estimator (ignoring correlations) Optimal approach if zero correlations (or linear decorrelation) Otherwise: significant performance loss Technical challenge: how to estimate the PDF shapes 3 ways: parametric fitting (function) Difficult to automate for arbitrary PDFs event counting nonparametric fitting Automatic, unbiased, Easy to automate, can create artefacts/suppress information but suboptimal TMVA uses binned shape interpolation using spline functions Or: unbinned adaptive Gaussian kernel density estimation TMVA performs automatic validation of goodness-of-fit LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 18

Multidimensional PDE Approach Use a single PDF per event class (sig, bkg), which spans Nvar dimensions 2 different implementations of the same idea: PDE Range-Search: count number of signal and background events in “vicinity” of test event preset or adaptive volume defines “vicinity” Improve simple event counting by use of kernel functions to penalise large distance from test event x 2 Carli-Koblitz, NIM A 501, 576 (2003) H 1 Increase counting speed with binary search tree test event Genuine k-NN algorithm (author R. Ospanov) Intrinsically adaptive approach H 0 Very fast search with kd-tree event sorting PDE-Foam: self adaptive phase space binning divides space in hyperqubes as “volumes”… x 1 Regression: assume the function value distance weighted averaged over the “volume” LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 19

Fisher’s Linear Discriminant Analysis (LDA) Well known, simple and elegant classifier LDA determines axis in the input variable hyperspace such that a projection of events onto this axis pushes signal and background as far away from each other as possible projection axis Classifier response couldn’t be simpler: “Fisher coefficients” Function discriminant analysis (FDA) Fit any user-defined function of input variables requiring that signal events return 1 and background 0 Parameter fitting: Genetics Alg. , MINUIT, MC and combinations Easy reproduction of Fisher result, but can add nonlinearities Very transparent discriminator LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 20

Nonlinear Analysis: Artificial Neural Networks Feed-forward Multilayer Perceptron Achieve nonlinear classifier response by “activating” output nodes using nonlinear weights 1 input layer 1 . . . Nvar discriminating input variables k hidden layers 1 . . . i j N M 1 . . 1 ouput layer 1 . . . 2 output classes (signal and background) Mk (“Activation” function) with: Three different implementations in TMVA (all are Multilayer Perceptrons) TMlp. ANN: Interface to ROOT’s MLP implementation MLP: TMVA’s own MLP implementation for increased speed and flexibility CFMlp. ANN: ALEPH’s Higgs search ANN, translated from FORTRAN Regression: uses the real valued output of ONE node LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 21

Boosted Decision Trees (BDT) DT: Sequential application of cuts splits the data into nodes, where the final nodes (leafs) classify an event as signal or background BDT: combine forest of DTs, with differently weighted events in each tree (trees can also be weighted) e. g. , “Ada. Boost”: incorrectly classified events receive larger weight in next DT Gradient. Boost “Bagging”: random poisson event weights resampling with replacement “Randomised Forest”, like bagging and usaging of random sub-sets of variables Boosting/bagging/randomising create set of “basis functions”: final classifier is linear combination of these improves stability Bottom-up “pruning” of a decision tree Remove statistically insignificant nodes to reduce tree overtraining LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 22

Boosted Decision Trees (BDT) DT: Sequential application of cuts splits the data into nodes, where the final nodes (leafs) classify an event as signal or background Decision tree after pruning Decision tree before pruning Bottom-up “pruning” of a decision tree Remove statistically insignificant nodes to reduce tree overtraining Randomised Forests don’t need pruning VERY ROBUST Some people even claim that any “boosted” decision trees don’t need to be pruned, but according to our experience, this is not always true. Regression: assume the function value of average in the leaf node LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 23

Learning with Rule Ensembles § Following Rule. Fit approach by Friedman-Popescu § Model is linear combination of rules, where a rule is a sequence of cuts Rule. Fit classifier rules (cut sequence rm=1 if all cuts satisfied, =0 otherwise) Sum of rules § normalised discriminating event variables Linear Fisher term The problem to solve is § § § Friedman-Popescu, Tech Rep, Stat. Dpt, Stanford U. , 2003 Create rule ensemble: use forest of decision trees Fit coefficients am, bk: gradient direct regularization minimising Risk (Friedman et al. ) Pruning removes topologically equal rules” (same variables in cut sequence) One of the elementary cellular automaton rules (Wolfram 1983, 2002). It specifies the next color in a cell, depending on its color and its immediate neighbors. Its rule outcomes are encoded in the binary representation 30=00011110 2. LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 24

Boosting Training Sample classifier C(0)(x) re-weight Weighted Sample re-weight classifier C(1)(x) Weighted Sample re-weight classifier C(2)(x) Weighted Sample re-weight classifier C(3)(x) Weighted Sample classifier C(m)(x) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 25

Adaptive Boosting (Ada. Boost) Training Sample classifier C(0)(x) § Ada. Boost re-weights events misclassified by previous classifier by: re-weight Weighted Sample re-weight classifier C(1)(x) Weighted Sample re-weight classifier C(2)(x) Weighted Sample re-weight classifier C(3)(x) Weighted Sample classifier C(m)(x) LHCb MVA Workshop, CERN, July 10, 2009 § Ada. Boost weights the classifiers also using the error rate of the individual classifier according to: H. Voss ― Multivariate Analysis with TMVA 26

Ada. Boost: A simple demonstration The example: (somewhat artificial…but nice for demonstration) : • Data file with three “bumps” var(i) > x • Weak classifier (i. e. one single simple “cut” ↔ decision tree stumps ) B b) var(i) <= x S a) Two reasonable cuts: a) Var 0 > 0. 5 εsignal=66% εbkg ≈ 0% misclassified events in total 16. 5% or b) Var 0 < -0. 5 εsignal=33% εbkg ≈ 0% misclassified events in total 33% the training of a single decision tree stump will find “cut a)” LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 27

Ada. Boost: A simple demonstration The first “tree”, choosing cut a) will give an error fraction: err = 0. 165 before building the next “tree”: weight wrong classified training events by ( 1 -err/err) ) ≈ 5 the next “tree” sees essentially the following data sample: re-weight b) . . and hence will chose: “cut b)”: Var 0 < -0. 5 The combined classifier: Tree 1 + Tree 2 the (weighted) average of the response to a test event from both trees is able to separate signal from background as good as one would expect from the most powerful classifier LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 28

Ada. Boost: A simple demonstration Only 1 tree “stump” LHCb MVA Workshop, CERN, July 10, 2009 Only 2 tree “stumps” with Ada. Boost H. Voss ― Multivariate Analysis with TMVA 29

Boosting Fisher Discriminant • Fisher Discriminant: • y(x) = A linear combination of the input variables • A linear combination of Fisher discriminant is again a Fisher discriminant. Hence “simple boosting doesn’t help” • However: • apply the “Fisher Discriminant” CUT in each step (or any other non-linear transformation” then it works! LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 30

Support Vector Machine There are methods to create linear decision boundaries using only measures of distances leads to quadratic optimisation processs • Using variable transformations, i. e moving into a higher dimensional space (i. e. using var 1*var 1 in Fisher Discriminant) allows in principle to separate with linear decision boundaries non linear problems • The decision boundary in the end is defined only by training events that are closest to the boundary • Support Vector Machine LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 31

Support Vector Machines Find hyperplane that best separates signal from background § x 2 Best separation: maximum distance (margin) between closest events (support) to hyperplane § Linear decision boundary § If data non-separable add misclassification cost parameter C· i i to minimisation function §Solution of largest margin depends only on Non-separable Separable data § support vectors op tim 1 a 2 l hyp e 3 rp la ne 4 margin inner product of support vectors (distances) x 1 quadratic minimisation problem LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 32

Support Vector Machines Find hyperplane that best separates signal from background § Best separation: maximum distance (margin) between closest events (support) to hyperplane § Linear decision boundary § If data non-separable add misclassification cost parameter C· i i to minimisation function x 3 Non-separable Separable data § x 2 (x 1, x 2) §Solution depends only on inner product of support vectors quadratic minimisation problem § x 1 Non-linear cases: § Transform variables into higher dimensional feature space where again a linear boundary (hyperplane) can separate the data § Explicit transformation doesn’t need to be specified. Only need the “scalar product” (inner product) x·x Ф(x)·Ф(x). § certain Kernel Functions can be interpreted as scalar products between transformed vectors in the higher dimensional feature space. e. g. : Gaussian, Polynomial, Sigmoid § Choose Kernel and fit the hyperplane using the linear techniques developed above Kernel size paramter typically needs careful tuning! (Overtraining!) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 33

Data Preparation and Preprocessing Data input format: ROOT TTree or ASCII Supports selection of any subset or combination or function of available variables Supports application of pre-selection cuts (possibly independent for signal and bkg) Supports global event weights for signal or background input files Supports use of any input variable as individual event weight Supports various methods for splitting into training and test samples: Block wise Randomly Periodically (i. e. periodically 3 testing ev. , 2 training ev. , 3 testing ev, 2 training ev. …. ) User defined separate Training and Testing trees Preprocessing of input variables (e. g. , decorrelation) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 34

Code Flow for Training and Application Phases Scripts can be ROOT scripts, C++ executables or python scripts (via Py. ROOT) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA T MVA tutorial 35

MVA Evaluation Framework TMVA is not only a collection of classifiers, but an MVA framework After training, TMVA provides ROOT evaluation scripts (through GUI) Plot all signal (S) and background (B) input variables with and without pre-processing Correlation scatters and linear coefficients for S & B Classifier outputs (S & B) for test and training samples (spot overtraining) Classifier Rarity distribution Classifier significance with optimal cuts B rejection versus S efficiency Classifier-specific plots: • Likelihood reference distributions • Classifier PDFs (for probability output and Rarity) • Network architecture, weights and convergence • Rule Fitting analysis plots • Visualisation of decision trees LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 36

Evaluating the Classifier Training Projective likelihood PDFs, MLP training, BDTs, … average no. of nodes before/after pruning: 4193 / 968 LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 37

Evaluating the Classifier Training Classifier output distributions for test and training samples … LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 38

Evaluating the Classifier Training Optimal cut for each classifiers … Using the TMVA graphical output one can determine the optimal cut on a classifier output (working point in “statistical significance) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 39

Evaluating the Classifier Training Background rejection versus signal efficiencies … Best plot to compare classifier performance If background in data non-uniform problem in training sample LHCb MVA Workshop, CERN, July 10, 2009 An elegant variable is the Rarity: transforms to uniform background. Height of signal peak direct measure of classifier performance H. Voss ― Multivariate Analysis with TMVA 40

Evaluating the Classifiers (taken from TMVA output…) Better variable Input Variable Ranking ---------- Fisher Fisher Fisher : : : : : Ranking result (top variable is best ranked) ----------------------Rank : Variable : Discr. power ----------------------1 : var 4 : 2. 175 e-01 2 : var 3 : 1. 718 e-01 3 : var 1 : 9. 549 e-02 4 : var 2 : 2. 841 e-02 ----------------------- How discriminating is a variable ? Classifier correlation and overlap ------- Factory Factory : : : Inter-MVA overlap matrix (signal): ---------------Likelihood Fisher Likelihood: +1. 000 +0. 667 Fisher: +0. 667 +1. 000 --------------- Do classifiers select the same events as signal and background ? If not, there is something to gain ! LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 41

Better classifier Evaluating the Classifiers (taken from TMVA output…) Check for overtraining Evaluation results ranked by best signal efficiency and purity (area) ---------------------------------------MVA Signal efficiency at bkg eff. (error): | Sepa. Signifi. Methods: @B=0. 01 @B=0. 10 @B=0. 30 Area | ration: cance: ---------------------------------------Fisher : 0. 268(03) 0. 653(03) 0. 873(02) 0. 882 | 0. 444 1. 189 MLP : 0. 266(03) 0. 656(03) 0. 873(02) 0. 882 | 0. 444 1. 260 Likelihood. D : 0. 259(03) 0. 649(03) 0. 871(02) 0. 880 | 0. 441 1. 251 PDERS : 0. 223(03) 0. 628(03) 0. 861(02) 0. 870 | 0. 417 1. 192 Rule. Fit : 0. 196(03) 0. 607(03) 0. 845(02) 0. 859 | 0. 390 1. 092 HMatrix : 0. 058(01) 0. 622(03) 0. 868(02) 0. 855 | 0. 410 1. 093 BDT : 0. 154(02) 0. 594(04) 0. 838(03) 0. 852 | 0. 380 1. 099 Cuts. GA : 0. 109(02) 1. 000(00) 0. 717(03) 0. 784 | 0. 000 Likelihood : 0. 086(02) 0. 387(03) 0. 677(03) 0. 757 | 0. 199 0. 682 ---------------------------------------Testing efficiency compared to training efficiency (overtraining check) ---------------------------------------MVA Signal efficiency: from test sample (from traing sample) Methods: @B=0. 01 @B=0. 10 @B=0. 30 ---------------------------------------Fisher : 0. 268 (0. 275) 0. 653 (0. 658) 0. 873 (0. 873) MLP : 0. 266 (0. 278) 0. 656 (0. 658) 0. 873 (0. 873) Likelihood. D : 0. 259 (0. 273) 0. 649 (0. 657) 0. 871 (0. 872) PDERS : 0. 223 (0. 389) 0. 628 (0. 691) 0. 861 (0. 881) Rule. Fit : 0. 196 (0. 198) 0. 607 (0. 616) 0. 845 (0. 848) HMatrix : 0. 058 (0. 060) 0. 622 (0. 623) 0. 868 (0. 868) BDT : 0. 154 (0. 268) 0. 594 (0. 736) 0. 838 (0. 911) Cuts. GA : 0. 109 (0. 123) 1. 000 (0. 424) 0. 717 (0. 715) Likelihood : 0. 086 (0. 092) 0. 387 (0. 379) 0. 677 (0. 677) --------------------------------------- LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 42

Subjective Summary of TMVA Classifier Properties Classifiers Criteria Performance no / linear correlations nonlinear correlations Training Speed Response Robust -ness Overtraining Weak input variables Curse of dimensionality Transparency Cuts Likelihood PDERS / k-NN / H-Matrix Fisher MLP BDT Rule fitting SVM The properties of the Function discriminant (FDA) depend on the chosen function LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 43

Regression Example: Target as a quadratic function of 2 variables • approximated by: • “Linear Regressor” • Neural Network LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 44

web page with quick reference summary of all training options! NEW! TMVA Users Guide for 4. 0 Available on http: //tmva. sf. net TMVA Users Guide 97 pp, incl. code examples ar. Xiv physics/0703039 LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 45

Some T o y E x a m p l e s LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 46

The “Schachbrett” Toy (chess board) Event distribution Performance achieved without parameter tuning: PDERS and BDT best “out of the box” classifiers After some parameter tuning, also SVM und ANN(MLP) perform equally well Theoretical maximum Events weighted by SVM response LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 47

More Toys: Linear-, Cross-, Circular Correlations Illustrate the behaviour of linear and nonlinear classifiers Linear correlations Circular correlations (same for signal and background) (opposite for signal and background) (same for signal and background) LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 48

How does linear decorrelation affect strongly nonlinear cases ? Original correlations SQRT decorrelation PCA decorrelation LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 49

Weight Variables by Classifier Output How well do the classifier resolve the various correlation patterns ? Linear correlations Cross-linear correlations Circular correlations (same for signal and background) (opposite for signal and background) (same for signal and background) Likelihood PDERS Fisher MLP BDT - D LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 50

Final Classifier Performance Background rejection versus signal efficiency curve: Linear Circular Cross Example LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 51

Stability with Respect to Irrelevant Variables Toy example with 2 discriminating and 4 non-discriminating variables ? LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 52

Stability with Respect to Irrelevant Variables Toy example with 2 discriminating and 4 non-discriminating variables ? use only two discriminant variables in classifiers LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 53

Stability with Respect to Irrelevant Variables Toy example with 2 discriminating and 4 non-discriminating variables ? use all only discriminant two discriminant variables in classifiers LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 54

general remarks about LHCb MVA Workshop, CERN, July 10, 2009 and systematics H. Voss ― Multivariate Analysis with TMVA 55

General Advice for (MVA) Analyses There is no magic in MVA-Methods: no need to be too afraid of “black boxes” you typically still need to make careful tuning and do some “hard work” The most important thing at the start is finding good observables good separation power between S and B little correlations amongst each other no correlation with the parameters you try to measure in your signal sample! Think also about possible combination of variables can this eliminate correlation Always apply straightforward preselection cuts and let the MVA only do the rest. “Sharp features should be avoided” numerical problems, loss of information when binning is applied simple variable transformations (i. e. log(var 1) ) can often smooth out these areas and allow signal and background differences to appear in a clearer way Treat regions in the detector that have different features “independent” can introduce correlations where otherwise the variables would be uncorrelated! Helge Voss Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 56

Some Words about Systematic Errors § Typical worries are: § What happens if the estimated “Probability Density” is wrong ? § Can the Classifier, i. e. the discrimination function y(x), introduce systematic uncertainties? § What happens if the training data do not match “reality” Any wrong PDF leads to imperfect discrimination function Imperfect ! (calling it “wrong” wouldn’t “right”) y(x) loss of discrimination power that’s all! classical cuts face exactly the same problem, But: in addition to cutting on features that are not correct, now you can also “exploit” correlations that are in fact not correct HOWEVER: see next slide § Systematic error are only introduced once “Monte Carlo events” with imperfect modeling are used for § efficiency; purity §#expected events § § same problem with classical “cut” analysis § use control samples to test MVA-output distribution (y(x)) Combined variable (MVA-output, y(x)) might “hide” problems in ONE individual variable train classifier with few variables only and compare with data check all input variables individually ANYWAY Helge Voss Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 57

Systematic “Error” in Correlations • Use as training sample events that have correlatetions • optimize CUTs • train an propper MVA (e. g. Likelihood, BDT) • Assume in “real data” there are NO correlations SEE what happens!! Helge Voss Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 58

Systematic “Error” in Correlations • Compare “Data” (Test. Sample) and Monte-Carlo (both taken from the same underlying distribution) Helge Voss Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 59

Systematic “Error” in Correlations • Compare “Data” (Test. Sample) and Monte-Carlo (both taken from the same underlying distributions that differ by the correlation!!! ) Differences are ONLY visible in the MVA-output plots Helge Voss (and if you’d look at cut sequences…. ) Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 60

Treatment of Systematic Uncertainties § Is there a strategy however to become ‘less sensitive’ to possible systematic uncertainties § i. e. classically: variable that is prone to uncertainties do not cut in the region of steepest gradient § classically one would not choose the most important cut on an uncertain variable § Try to make classifier less sensitive to “uncertain variables” § i. e. re-weight events in training to decrease separation in variables with large systematic uncertainty (certainly not yet a recipe that can strictly be followed, more an idea of what could perhaps be done) Helge Voss “Calibration uncertainty” may shift the central value and hence worsen (or increase) the discrimination power of “var 4” Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 61

Classifier output distributions for signal only Treatment of Systematic Uncertainties Helge Voss 1 st Way Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 62

Classifier output distributions for signal only Treatment of Systematic Uncertainties Helge Voss 2 nd Way Graduierten-Kolleg, Freiburg, 11. -15. Mai 2009 ― Multivariate Data Analysis and Machine Learning 63

Copyrights & Credits TMVA is open source software Use & redistribution of source permitted according to terms in BSD license Acknowledgments: The fast development of TMVA would not have been possible without the contribution and feedback from many developers and users to whom we are indebted. Andreas Hoecker (CERN, Switzerland), Jörg Stelzer (CERN, Switzerland), Peter Speckmayer (CERN, Switzerland), Jan Therhaag (Universität Bonn, Germany), Eckhard von Toerne (Universität Bonn, Germany), Helge Voss (MPI für Kernphysik Heidelberg, Germany), Tancredi Carli (CERN, Switzerland), Asen Christov (Universität Freiburg, Germany), Or Cohen (CERN, Switzerland Weizmann, Israel), Krzysztof Danielowski (IFJ and AGH/UJ, Krakow, Poland), Dominik Dannheim (CERN, Switzerland), Sophie Henrot-Versille (LAL Orsay, France), Matthew Jachowski (Stanford University, USA), Kamil Kraszewski (IFJ and AGH/UJ, Krakow, Poland), Attila Krasznahorkay Jr. (CERN, Switzerland, and Manchester U. , UK), Maciej Kruk (IFJ and AGH/UJ, Krakow, Poland), Yair Mahalalel (Tel Aviv University, Israel), Rustem Ospanov (University of Texas, USA), Xavier Prudent (LAPP Annecy, France), Arnaud Robert (LPNHE Paris, France), Doug Schouten (S. Fraser U. , Canada), Fredrik Tegenfeldt (Iowa University, USA, until Aug 2007), Alexander Voigt (CERN, Switzerland), Kai Voss (University of Victoria, Canada), Marcin Wolter (IFJ PAN Krakow, Poland), Andrzej Zemla (IFJ PAN Krakow, Poland). Backup slides on: (i) some illustrative toy examples (ii) treatment of systematic uncertainties (iii) sensitivity to weak input variables http: //tmva. sf. net/ LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 64

Questions/Discussion points How to treat systematic uncertainties in MVA methods? check multidimensioal training samples including their correlations which differences we care about? Can one give a particular classifier that is: always the best? always the best for specific class of models? How could one teach the classifiers to optimise for certain regions of interest (i. e. very large background rejection only ? ) is that actually necessary, or does the cut on the MVA output provide just this? Train according to “smallest error on measurement” rather than best separation of signal/background Anyone a good idea for maybe better treatment of negative weights? How to best divide the data sample into “test and training” sample? Do we want an additional validation sample? automatic tuning of parameters cross validation LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 65

Bias Variance Trade off -- Overtraining low bias low variance high variance prediction error high bias test sample Overtraining! training sample low high model complexity x 2 S B B x 1 LHCb MVA Workshop, CERN, July 10, 2009 S H. Voss ― Multivariate Analysis with TMVA x 1 66

Cross Validation § many (all) classifiers have tuning parameters “a” that need to be controlled against overtraining § number of training cycles, number of nodes (neural net) § smoothing parameter h (kernel density estimator) §…. § the more free parameter a classifier has to adjust internally more prone to overtraining § more training data better training results § division of data set into “training” and “test” sample reduces the “training” data Cross Validation: divide the data sample into say 5 sub-sets Test Train Test Train Train § train 5 classifiers: yi(x, a) : i=1, . . 5, § classifier yi(x, a) is trained without the i-th sub sample § calculate the test error: § choose tuning parameter a for which CV(a) is minimum and train the final classifier using all data LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 67

Minimisation Robust global minimum finder needed at various places in TMVA Brute force method: Monte Carlo Sampling Sample entire solution space, and chose solution providing minimum estimator Good global minimum finder, but poor accuracy Default solution in HEP: (T)Minuit/Migrad [ How much longer do we need to suffer …. ? ] Gradient-driven search, using variable metric, can use quadratic Newton-type solution Poor global minimum finder, gets quickly stuck in presence of local minima Specific global optimisers implemented in TMVA: Genetic Algorithm: biology-inspired optimisation algorithm Simulated Annealing: slow “cooling” of system to avoid “freezing” in local solution TMVA allows to chain minimisers For example, one can use MC sampling to detect the vicinity of a global minimum, and then use Minuit to accurately converge to it. LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 68

Minimisation Techniques Monte Carlo Brute force methods: Random Monte Carlo or Grid search Grid Search • Sample entire solution space, and chose solution providing minimum estimator • Good global minimum finder, but poor accuracy Default solution in HEP: Minuit • Gradient-driven search, using variable metric, can use quadratic Newton-type solution • Poor global minimum finder, gets quickly stuck in presence of local minima Genetic Algorithm Biology-inspired • “Genetic” representation of points in the parameter space • Uses mutation and “crossover” • Finds approximately global minima Simulated Annealing Like heating up metal and slowly cooling it down (“annealing”) Atoms in metal move towards the state of lowest energy while for sudden cooling atoms tend to freeze in intermediate higher energy states slow “cooling” of system to avoid “freezing” in local solution LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 69

Binary Trees Tree data structure in which each node has at most two children Typically the child nodes are called left and right Binary trees are used in TMVA to implement binary search trees and decision trees Root node Depth of a tree < > Amount of computing time to search for events: • Box search: (Nevents)Nvar • BT search: Nevents·Nvarln 2(Nevents) Leaf node LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 70

Gauss-Decorr-Gauss etc. . G GDGDGDG LHCb MVA Workshop, CERN, July 10, 2009 H. Voss ― Multivariate Analysis with TMVA 71