Higgs Machine Learning Challenge Claire AdamBourdarios Glen Cowan

Higgs Machine Learning Challenge Claire Adam-Bourdarios, Glen Cowan, Cécile Germain, Isabelle Guyon, Balazs Kegl, David Rousseau higgsml@lal. in 2 p 3. fr http: //higgsml. lal. in 2 p 3. fr Higgs. ML visits CERN, 19 th May 2015

Outline q. Machine Learning, Challenges … q. The Higgs Machine Learning challenge q. What’s next David Rousseau Higgs. ML visits CERN, 19 th May 2015 2

Machine Learning and HEP q Machine Learning is more or less what we call Multi Variate Analysis q Neural Nets used somewhat in the 90’ies (e. g. LEP), Fisher Discriminant as well q BDT (Adaboost) invented in 97 q MVA techniques have been used extensively at D 0/CDF (mostly BDT, but not only) in the 00’ies q ATLAS/CMS less eager to adopt MVA at LHC starts for some good reasons: o o Need to understand well the input variables first Still a lot to gain by improving input variables Systematics more difficult to evaluate Collected luminosity was increasing fast q But lot of work recently with MVA techniques o Competition o Best use of available data q Meanwhile Neural Net re-appear in their “deep” incarnation David Rousseau Higgs. ML visits CERN, 19 th May 2015 3

Machine Learning and HEP (2) q However: o TMVA, within Root, has been instrumental in popularising MVA technique within HEP o Most HEP people using TMVA, most people using BDT in TMVA o Although getting a reasonable answer from TMVA is quick and easy, it takes time to really become an expert with e. g. BDT o People are focussing on the choices of input variables and the evaluation of systematics (which of course are excellent things to do) q Not much work within LHC experiments on studying possible better MVA techniques, for which you need the software and the know-how q Enormous development of Machine Learning in the outside world in the last 10 years (“Big Data”, “Data Science” buzz words, even “Artificial Intelligence” is back) David Rousseau Higgs. ML visits CERN, 19 th May 2015 4

Machine Learning Challenge ? q Challenges have become in the last 10 years a common way of working for the machine learning community q Machine learning scientists are eager to test their algorithms on real life problems more valuable(=publisheable) than artificial problems q Company or academics want to outsource a problem to machine learning scientist, but also geeks etc. q Some companies make a business from organising challenges: kaggle q A few recent examples now… David Rousseau Higgs. ML visits CERN, 19 th May 2015 5

Looking at People (2012 -14) Actions Wave Point Interactions Clap Shake Hands Hug http: //chalearn. org/ David Rousseau Fight Higgs. ML visits CERN, 19 th May 2015 6

Neural connectomics (2015) http: //chalearn. org/ David Rousseau Higgs. ML visits CERN, 19 th May 2015 7

Genomics Challenge submissions David Rousseau Higgs. ML and tracking challenges Best in-house solutions CTD 2015 Berkeley 8

NEW: Auto. ML challenge (2015) Fully automatic machine learning without ANY human intervention http: //codalab. org/Auto. ML December 2014 – May 2015 $30, 000 in prizes David Rousseau Higgs. ML visits CERN, 19 th May 2015 9

Some problems in the outside world look like what we are doing in HEP

Classification (obviously) q. Facebook challenge on Kaggle (https: //www. kaggle. com/c/facebook-recruiting-ivhuman-or-bot, ends 8 th June 2015) o Predict if an online bid is made by a machine or a human David Rousseau Higgs. ML visits CERN, 19 th May 2015 11

Something like tracking q one example among many from NIPS 2014 : http: //papers. nips. cc/paper/5572 -a-complete-variational-tracker. pdf Also: q Note that these are real-time applications, with CPU constraints q Worry about efficiency, “track swap” David Rousseau Higgs. ML visits CERN, 19 th May 2015 12

Something like trigger Real-time face recognition : efficiency, fake, CPU time… David Rousseau Higgs. ML visits CERN, 19 th May 2015 13

Why challenges work ? Not just ML, but a general trend: Open Innovation David Rousseau Higgs. ML visits CERN, 19 th May 2015 14

From domain to challenge and back Domain e. g. HEP Problem Challenge organisation simplify Challenge Problem The crowd solves the challenge problem Domain experts solve the domain problem reimport Solution David Rousseau Higgs. ML visits CERN, 19 th May 2015 Solution 15

Higgs Machine Learning Challenge

… in a nutshell q Why not put some ATLAS simulated data on the web and ask data scientists to find the best machine learning algorithm to find the Higgs ? o Instead of HEP people browsing machine learning papers, coding or downloading possibly interesting algorithm, trying and seeing whether it can work for our problems q Challenge for us : make a full ATLAS Higgs analysis simple for non physicists, but not too simple so that it remains useful q Also try to foster long term collaborations between HEP and ML David Rousseau Higgs. ML visits CERN, 19 th May 2015 17

Committees q Organization committee: Machine Learning ATLAS { { o o o Claire Adam-Bourdarios : ATLAS-LAL Glen Cowan : ATLAS-RHUL David Rousseau : ATLAS-LAL Cécile Germain : TAO-LRI Isabelle Guyon : Chalearn (challenges organisation) Balazs Kegl : Appstat-LAL q Advisory committee: o o Andreas Hoecker : ATLAS-CERN Marc Schoenauer : INRIA Joerg Stelzer : ATLAS-CERN Thorsten Wengler : ATLAS-CERN David Rousseau Higgs. ML visits CERN, 19 th May 2015 18

H tautau ATLAS-CONF-2013 -108 4. 1 s evidence (now superseded by paper ar. Xiv: 1501. 049 tbp in JHEP. Analysis methods essentially unchanged. ) David Rousseau Higgs. ML visits CERN, 19 th May 2015 19

Dataset ASCII csv file, with mixture of Higgs to tautau (lephad) signal and corresponding backgrounds, from official GEANT 4 ATLAS simulation Weight and signal/background (for training dataset only) weight (fully normalised) label : « s » or « b » Conf note variables used for categorization or BDT: DER_mass_MMC DER_mass_transverse_met_lep DER_mass_vis DER_pt_h DER_deltaeta_jet VBF DER_mass_jet signature DER_prodeta_jet DER_deltar_tau_lep DER_pt_tot DER_sum_pt DER_pt_ratio_lep_tau DER_met_phi_centrality DER_lep_eta_centrality } David Rousseau Primitive 3 -vectors allowing to compute the conf note variables (mass neglected), 16 independent variables: PRI_tau_pt PRI_tau_eta PRI_tau_phi PRI_lep_pt PRI_lep_eta PRI_lep_phi PRI_met_phi PRI_met_sumet PRI_jet_num (0, 1, 2, 3, capped at 3) PRI_jet_leading_pt PRI_jet_leading_eta PRI_jet_leading_phi PRI_jet_subleading_pt PRI_jet_subleading_eta PRI_jet_subleading_phi PRI_jet_all_pt Higgs. ML visits CERN, 19 th May 2015 } VBF signature 20

Significance q Need to have one robust estimator of the quality of the classification algorithm q Decided to use the well known (in HEP) “Asimov” formula (G. Cowan, K. Cranmer, E. Gross, and O. Vitells, “Asymptotic formulae for likelihood-based tests of new physics”, EPJCC, vol. 71, pp. 1– 19, 2011. ) with regularization on top o √(2*((s+b’)*log(1+s/b’)-s)) ~s/√b’ o with s and b’=b+10 normalised to 2012 data taking luminosity: o s=S(selected signal) weights_i o b=S(selected background) weights_i q Why b’=b+10 (“regularisation”) : practical way to avoid large significance fluctuation when small phase space region with very few background events is chosen. Do not want to pick winners on their luck. q Note that normalisation already included in the weights : no need to explain integrated luminosity and cross-section q Glen Cowan has derived a new version of Asimov formula including a sigma_b from systematics or statistics However in our case this leads to favour small David Rousseau region large variance. Higgs. ML visits CERN, 19 th May 2015 21

How did it work ? q First idea in Sep 2012 q Challenge ran from May to September 2014 q People register to Kaggle web site hosted https: //www. kaggle. com/c/higgs-boson. (additional info on https: //higgsml. lal. in 2 p 3. fr) q Open to almost any one o o Data scientist HEP physicists Students, geeks, Except LAL-Orsay employees (for legal reasons) …download training dataset (with label) with 250 k events …train their own algorithm to optimise the significance (à la s/sqrt(b)) …download test dataset (without labels) with 550 k events …upload their own classification The site automatically calculates significance. Public (100 k events) and private (450 k events) leader boards update instantly. (Only the public is visible) q Competition closed mid september 2014. Private leaderboard is disclosed. People are asked to provide their code and methods. Best 1 2 3 win 7 k$ 4 k$ 2 k$ q In addition, the potentially most interesting one gets the “HEP meets ML award” q q q Funded by: Paris Saclay Center for Data Science, Google, INRIA David Rousseau Higgs. ML visits CERN, 19 th May 2015 22

From domain to challenge and back Domain e. g. HEP Challenge organisation Challenge 18 months Problem simplify Problem The crowd solves the challenge problem Domain experts solve the domain problem reimport Solution David Rousseau Higgs. ML visits CERN, 19 th May 2015 Solution 23

Real analysis vs challenge 1. 2. Systematics (and data vs MC) 2 categories x n BDT score bins 1. 2. 1. Background estimated from data (embedded, anti tau, control region) and some MC Weights include all corrections. Some negative weights (tt) Potentially use any information from all 2012 data and MC events Few variables fed in two BDT 3. 4. 2. 3. 4. 1. 2. Significance from complete fit with NP etc… MVA with TMVA BDT 5. 6. 7. 8. No systematics No categories, one signal region Straight use of ATLAS G 4 MC Weights only include normalisation and pythia weight. Neg. weight events rejected. Only use variables and events preselected by the real analysis All BDT variables + categorisation variables + primitives 3 -vector Significance from “regularised Asimov” MVA “no-limit” Simpler, but not too simple! David Rousseau Higgs. ML visits CERN, 19 th May 2015 24

Participation q. Big success ! q 1785 teams (1942 people) have participated (participation=submission of at least one solution) o (6517 people have downloaded the data) o most popular challenge on the Kaggle platform (until a few weeks ago) o 35772 solutions uploaded q 136 forum topics with 1100 posts David Rousseau Higgs. ML visits CERN, 19 th May 2015 25

Final leaderboard 7000$ 4000$ 2000$ Best physicist HEP meets ML award XGBoost authors Free trip to CERN David Rousseau Higgs. ML visits CERN, 19 th May 2015 26

Meet the winners q See http: //atlas. ch/news/2014/machine-learning-wins-the -higgs-challenge. html q 1 : Gabor Melis (Hungary) sw developer and consultant : wins 7000$. 3 rd talk. q 2 : Tim Salimans (Neitherland) data science consultant: wins 4000$ q 3 : Pierre Courtiol (nhlx 5 haze) (France) ? : wins 2000$ q HEP meets ML award: (team crowwork), Tianqi Chen (U of Washington Ph. D student in Data Science) and Tong He (graduate student Data Science SFU). Provided XGBoost used by many participants. Win a free trip and visit to CERN in 2015, this is today! 2 nd talk. David Rousseau Higgs. ML visits CERN, 19 th May 2015 ? 27

Rank distribution after bootstrap Distribution of rank of participant of rank i after 1000 bootstraps of the test sample. ! Gabor clearly better rank 1 rank 2 rank 4 rank 7 rank 5 rank 8 David Rousseau rank 3 rank 6 rank 9 Higgs. ML visits CERN, 19 th May 2015 28

=significance Private score vs public: Gabor David Rousseau Higgs. ML visits CERN, 19 th May 2015 29

=significance Private score vs public: Lubos Clear overtraining ! David Rousseau Higgs. ML visits CERN, 19 th May 2015 30

Clearly better! David Rousseau Higgs. ML visits CERN, 19 th May 2015 31

TMVA vs Gabor gabor VBF category tmva boosted category David Rousseau q vbf, boosted categories as is ATLAS note (no ATLAS insider information) q tmva, gabor are trained without categories, on full 30 variables (not directly comparable to ATLAS analysis) q (also significance is simple asimov, no bin, no systematics (and fake tau missing)) q Gabor improves more significantly in VBF categories (2 jets events more complex) Higgs. ML visits CERN, 19 th May 2015 32

Intermezzo

QM computation with ML q. One day NN…NLO computation with Machine Learning? David Rousseau Higgs. ML visits CERN, 19 th May 2015 Computed with hybrid Density-Functional Theory (accurate but computer intensive) 34

Outlook

What did we learn q Very successful satellite workshop at NIPS (on of the two major Machine Learning conferences) in Dec 2014 @ Montreal: https: //indico. lal. in 2 p 3. fr/event/2632/ q In short (much more in next two talks): o o o 20% gain w. r. t. to untuned TMVA deep Neural nets (but marginally better than BDT) rules Ensemble methods (random forest, boosting) rule Meta-ensembles of diverse models careful cross-validation (250 k training sample really small) Complex software suites using routinely multithreading, GPU, etc… (e. g. XGBoost, scikit-learn) o Some techniques (e. g. meta-ensembles) too complex to be practical, and marginal gain… David Rousseau Higgs. ML visits CERN, 19 th May 2015 o …Others appear practical and useful 36

From domain to challenge and back Domain e. g. HEP Challenge organisation Challenge 18 months Problem simplify Domain experts solve the domain problem 4 months The crowd solves the challenge problem >n months/years ? reimport Solution David Rousseau Higgs. ML visits CERN, 19 th May 2015 Solution 37

Outlook q Now working on re-importing into HEP all the ML developments q Dataset has been released on CERN Open Data Portal http: //opendata. cern. ch/collection/ATLAS-Higgs-Challenge-2014 , to remain available until the end of time (citeable with a d. o. i) o Release with the full truth q NIPS proceedings write-up (with detailed description of “how they did it ? ”) to be released before the summer q Many ideas/software/methodology to be digested and tried out possibly on different problems. Drop us a line higgsml@lal. in 2 p 3. fr if your are interested. q Mailing list just opened to any one with an interest in both Data Science and High Energy Physics : HEP-datascience@googlegroups. com q Workshop “Data Science @ LHC” at CERN, 9 th-13 th Nov. 2015 (under the umbrella of LHC Physics Center at CERN http: //lpcc. web. cern. ch/LPCC/ ) q Rumors about other HEP challenges being prepared David Rousseau Higgs. ML visits CERN, 19 th May 2015 38

Spares

Machine Learning? q Machine Learning is the part of computer science which, in particular, deals among other with automatic classification: o E. g. neural nets to read handwritten digits (~30 years ago) o In HEP we call it MVA (Multi Variate Analysis) q Developing rapidly o Lots of data to deal with o Lots of CPU power too o Big money involved: § Google/Facebook advertisements based on your searches, your gmail messages, your +1/likes § amazon : “we recommend for you” based on what you already bought o “Big data” buzz word o New field “Data Science” David Rousseau Higgs. ML visits CERN, 19 th May 2015 40

What make a good challenge? Isabelle Guyon http: //chalearn. org/ Good problem Good data + enough data Clear objective + good metric Simple rules + no Information Protection contingencies Prizes (~ $5000) Starter kit On-line feed-back Publication outlets Avoid many pitfalls, in particular leaking the truth, or a large luck factor in the ranking q (see http: //ciml. chalearn. org/schedule a NIPS satellite workshop on organising challenges, in particular Ben Hammer’s, Kaggle chief scientist, on do’s and don’ts) q q q q q David Rousseau Higgs. ML visits CERN, 19 th May 2015 41

What data did we release ? From ATLAS full sim Geant 4 MC 12 production 30 variables Signal is H tautau, Background a mixture of : Z, top, W Based on November 2013 ATLAS Htautau conf note ATLAS-CONF 2013 -108 q Preselection for lep-had topology : single lepton trigger, one lepton identified, one hadronic tau identified q 800. 000 events: q q o 250. 000 training data set o 550. 000 test data set without label and weight q Reproduces reasonably well (~20%) content of 3 highest sensitivity bins (x 2 categories) in conf note q (some background and many correction factors deliberately omitted so that the sample cannot be used for physics, only for machine learning studies) David Rousseau Higgs. ML visits CERN, 19 th May 2015 42

The ATLAS hacker q In principle, since the data we release is available to all ATLAS members, one ATLAS hacker could: o use the variables to match back (using lepton/hadron/jet Pt eta phi) to the original MC event o check whether it is signal or background o cheat to get best significance q (this would be discovered at the software release stage) q This was transparently explained to ATLAS : it would be bad for our image and counterproductive if the public leaderboard is cluttered by ATLAS hackers q no evidence/no hint there was such attempt David Rousseau Higgs. ML visits CERN, 19 th May 2015 43

ATLAS hacker (2) Relative smearing q We thought of smearing the parameters to prevent the matching q Back of an envelope calculation: by how much should D variables be smeared so that the original can be matched with 50% probability among N entries q smearing should be at least 15% q events become meaningless q bad idea! Not pursued D=16 D=3 David Rousseau D=8 Higgs. ML visits CERN, 19 th May 2015 N entries 44

ATLAS hacker (3) q Hiding the origin of an entry in a DB is called “sanitizing”, this is a notoriously difficult, and very hot topic, e. g: o Finding owner of a medical record given anonymized parameters (gender, zip code, and birth date uniquely identifies 83% of US people) o Finding owner of a mobile phone given hour and area of a few calls (Pour La Science June 2014): David Rousseau Higgs. ML visits CERN, 19 th May 2015 45

Licensing issues q Anyone participating to the challenge had to agree to the rules ( https: //www. kaggle. com/c/higgs-boson/rules ) in particular: q Software o Participants can use whatever software they like, but to win a price, they have to release it under an OS license (so that we can 1) verify it 2) use it eventually) q Simulated data: o All ATLAS real and simulated data belongs (legally) to CERN o We’ve had the signature of S. Bertolucci, CERN Director of Research, to release the data o Data was initially made available only for the duration of the challenge and only for the challenge (“Can I use the data for a master thesis ? ” “Sorry no. ”) finally agreement later to release the data on CERN ODP (got the same question again from someone else, “Yes please!”) David Rousseau Higgs. ML visits CERN, 19 th May 2015 46

Transfer Learning (2011) • Learn a data representation from one task: • Use it for another task: http: //chalearn. org/ David Rousseau Higgs. ML visits CERN, 19 th May 2015 47

Million $ prize challenges 2009 $1, 000 2012 David Rousseau Higgs. ML visits CERN, 19 th May 2015 48

BDT and NN q BDT (Boosted Decision Tree) which is by far the most used technique in Atlas/CMS is actually an old technique (Adaboost 1997) q More recent techniques, just an example: o Unsupervised neural network (Example: The Google cat: deep learning technique running on 16 K cores for three days, watching 10 M random You. Tube video stills [Le et al. , ICML’ 12] David Rousseau Higgs. ML visits CERN, 19 th May 2015 49

Are winning score different ? David Rousseau Higgs. ML visits CERN, 19 th May 2015 50

30 20 10 80 70 60 50 David Rousseau Tuned and improved tmva BDT Simple tmva BDT 40 3 3. 02 3. 04 3. 06 3. 08 3. 12 3. 14 3. 16 3. 18 3. 22 3. 24 3. 26 3. 28 3. 32 3. 34 3. 36 3. 38 3. 42 3. 44 3. 46 3. 48 3. 52 3. 54 3. 56 3. 58 3. 62 3. 64 3. 66 3. 68 3. 72 3. 74 3. 76 3. 78 3. 82 3. 84 3. 86 3. 88 3. 92 3. 94 3. 96 3. 98 4 Best private scores 0 score Higgs. ML challenge, GDR Terascale 2015, Saclay 51

Private score vs public: nhlx 5 haze Optimal! David Rousseau Higgs. ML visits CERN, 19 th May 2015 52