Examplestools for analysis on GPU Diego Martinez Santos

Examples/tools for analysis on GPU Diego Martinez Santos (GAIN / USC) 1

Introduction • It is known that Graphics cards (GPU’s) can be highly performing for tasks that are parallelizable into subtasks (eg, the card of a PS 4 reaches 1. 8 TFLOPS) 2

Introduction • It is known that Graphics cards (GPU’s) can be highly performing for tasks that are parallelizable into subtasks • For this reason, they are a potential architecture for future HLT’s • We had a GTX 980 Ti (that we purchased for HLT upgrade studies) 3

Introduction • It is known that Graphics cards (GPU’s) can be highly performing for tasks that are parallelizable into subtasks • For this reason, they are a potential architecture for future HLT’s • We had a GTX 980 Ti (that we purchased for HLT upgrade studies) • … But we also learned in UFRJ that the same devices can be highly performant for offline fits. (eg, Goo. Fit) 4

Introduction • In Santiago I started ~2 month ago to try the GTX for some cases in which the CPU time was going to be a problem • A μμ (upper limits based on millions of toys, with a model that requires numeric convolution, etc…) • Bs J/ψΦ : fit a massive J/ψKK MC tuple 5

Introduction • In Santiago we migrated most of our analyses to GPU • A μμ (upper limits based on millions of toys, with a model that requires numeric convolution, etc…) • Bs J/ψΦ : fit a massive J/ψKK MC tuple • B ->4 h analyses to GPU • K µµ 6

Introduction • So, I combined some python packages into a kind of very simple framework, to see how much speed can we gain • There is a version in git (2018, will update later this year): • I am not using neither Goo. Fit nor ROOT. We made custom FCN’s and pass them to Minuit Still, everything is consistent with ROOT, so can be used simultaneously. Works on Ubuntu, SLC and Cent. OS 7. Also managed to make it work in Windows but not efficiently. 7

Basic idea Matrix of data points CPU Binning (for numeric integral) (Minimizing I/O is very important here) Pdf(t, Ω | {fit_params}) Pdf(t, | {fit_params}) 8

Basic idea FCN Values for {fit_parameters} CPU (Minimizing I/O is very important here) Pdf(t, Ω | {fit_params}) Pdf(t, | {fit_params}) Data Binning for numeric integral 9

Basic idea FCN CPU (Minimizing I/O is very important here) • Likelihood for each data point • 1 D pdf for the time 10

Basic idea FCN Array of log likelihoods , {LL_i} (or the sum{LL_i} if the array is large) CPU 1 D time pdf as an array (Minimizing I/O is very important here) Pdf(t, Ω | {fit_params}) Pdf(t, | {fit_params}) Data Binning for numeric integral 11

Basic idea FCN CPU • Σ (LL_i) (if not done @GPU) • Σ (pdf) (normalization) FCN Minuit 12

Basic idea. Programing language(s) Python CPU CUDA-C Organic CPU I’ve converged towards : • Pdf’s implemented as __device__ void Cuda-C functions (stay in the GPU) • Likelihood evaluation/toy generation as __global__ • Fit framework/configuration/OO/etc in Python. (py. CUDA + python analysis software + custom classes for parameter bkk, etc…) 13

Basic idea. Programing language(s) Python CPU CUDA-C Organic CPU • Global/external constraints also at the python level (no point on parallelizing them): FCN += (( np – mean_np)/sigma_np) **2 14

Observed timings (A μμ) • First tests done for the A μμ analysis: • GPU seems very good at generated large samples of random (poisson) numbers (very useful for toy. MC’s used for limit setting): • 1 billion of random numbers takes 65 x less than in a CPU • The GPU is certainly cheaper than a 65 cores CPU 15

Observed timings (A μμ) • First tests done for the A μμ analysis: • GPU seems very good at generated large samples of random (poisson) numbers (very useful for toy. MC’s used for limit setting): • 1 billion of random numbers takes 65 x less than in a CPU • The GPU is certainly cheaper than a 65 cores CPU • The mass fit is also fairly complex (Binned fit with 100 k bins, Ypatia distribution –including the numeric convolution due to MS), numeric integral… Yet it takes ~5 s. Observed gains in A μμ go btwn ~60 x to even hundreds Worth investigating if GPU can deal with J/ψKK φs fit on massive NTuples (toy. MC) • Points Model 16

Observed timings J/ψ KK All parameters free, numeric integral for the time (even if not necessary). Call Migrad and Hesse (not Minos, otherwise the CPU-only fit takes ages) • 100 K events: • 1 KK mass bin ( 2 categories, 10 free params): < 4 sec. (771 calls) ~same in CPU (C++) : 520 sec ( > 100 x gain !) (530 calls) 17

Observed timings J/ψ KK All parameters free, numeric integral for the time (even if not necessary). Call Migrad and Hesse (not Minos, otherwise the CPU-only fit takes ages) • 100 K events: • 1 KK mass bin ( 2 categories, 10 free params): < 4 sec. (771 calls) ~same in CPU (C++) : 520 sec ( > 100 x gain !) (530 calls) • 10 M events: • 1 KK mass bin ( 2 categories, 10 free params): ~< 1 minute (612 calls) • 6 KK mass bins ( 12 categories, 20 free params): ~ 3. 5 min (2882 calls) (and it can probably be reduced by a better handling of the categories, and/or by using reals instead of complex. But at this point it doesn’t matter to me) 18

Our repository My experience is that one doesn’t need a complicated framework for any of these analyses, but in any case what we provide is: • Example of a simple fit without custom classes, just plain use of existing, fairly well known, python packages 19

Our repository My experience is that one doesn’t need a complicated framework for any of these analyses, but in any case what we provide is: • Example of a simple fit without custom classes, just plain use of existing, fairly well known, python packages • Some custom classes for bookkeping of parameters, categories for simultaneous fits, plotting, toy. MC, FFT interfacing to several fiting packages, … mycategory = manager. create. Fit() manager. fit. migrad() manager. fit. hesse() 20

Our repository My experience is that one doesn’t need a complicated framework for any of these analyses, but in any case what we provide is: • Example of a simple fit without custom classes, just plain use of existing, fairly well known, python packages • Some custom classes for bookkeping of parameters, categories for simultaneous fits, plotting, toy. MC, FFT interfacing to several fiting packages, … • Examples using the classes above • Simple CLs examples • A mm-like fit • Simple J/ψΚΚ fit, including toy. MC • Urania’s symbolic stuff 21

Our repository My experience is that one doesn’t need a complicated framework for any of these analyses, but in any case what we provide is: • Example of a simple fit without custom classes, just plain use of existing, fairly well known, python packages • Some custom classes for bookkeping of parameters, categories for simultaneous fits, plotting, toy. MC, FFT interfacing to several fiting packages, … • Examples using the classes above • Simple CLs examples • A mm-like fit • Simple J/ψΚΚ fit, including toy. MC • Urania’s symbolic stuff • git ready, together with some documentation 22

Observed timings B KπΚπ • The fit takes typically 20 h in a 20 cores machine 1 to converge • Migrated to GPU • Converged in 5 min • Breakdown of the gains: • ~1700 x total gain • ~60 x gain due to hardware (GPU vs CPU) –obtained by calculating the time per minuit call. • The other ~28 x seems to come from organic CPU: the way we implemented the FCN seems to avoid a nan and go directly to the right minimum, while the Minuit call through Roo. Fit implementation seems to have been jumping on local minima 1(two Xeon E 5 2680 v 3) 23

Add Symbolic stuff • In the same way as for Roo. Fit, one can use Urania to calculate pdf expressions in an automatic way, and print them into cuda code See https: //indico. cern. ch/event/288649/contributions/1642993/attachments/538618/742508/Sympy_in_Urania_v 2. pdf • Eg , time dependent angular distributions for an arbitrary number of waves • Analytical integrals • Analytical derivatives: (this can be interesting if you want to write a minimizer to substitute Minuit) 24

Summary Repository: git clone https: //gitlab. cern. ch/bsm-fleet/Ipanema. git (is going to be updated) . Documentation: https: //arxiv. org/abs/1706. 01420 • Examples of how to combine python packages for HEP analysis in CUDA • Maximun likelihood fits, CLs limits … • toy. MC generation • Bookkeeping classes to handle parameters, plots, and interfaces to • Minuit • Sampling ( via multinest package, others to be added) • The custom classes we created are fairly simple, you can inherit/modify/rewrite at your will 25

Conclusions • If you have a time consuming fit/limit setting code, and were wondering about using GPU’s: I think it is really worth • Shown great reduction on the timing for realistic analyses: • Found gains of 100 or even 1000 x • But gain from hardware seems ~~60 x for a GTX 980 Ti to a single core device • Implementing(thus understanding) your own problem from scratch gives the extra factor (up to 28 x) • Typically 10 x better performance per euro compared to CPU’s • Tested the following models (some available in cheap gaming laptops): GTX 980 Ti, GTX 1050, GTX 1060, GTX 1080, Tesla M 2090, GTX 1080 Ti, RTX 2080 Ti. 26