HSF E WLCG STRATEGY DOCUMENT Tommaso Boccali INFN

HSF E WLCG STRATEGY DOCUMENT Tommaso Boccali (INFN Pisa)

~ single event complexity / size … C H L f o % 3 d i d e W ~ ~ total integrated storage needed THE PROBLEM #1 HE-LHC

THE PROBLEM #2 • Why? Some numbers and resource scaling • Trigger rates 1 k. Hz 5 -10 k. Hz (factor 510 x) • More complex detector (more acquisition channels) + … • … + More crowded events (PU from 35 to 200) • Overall resource increase ”if we do nothing” • (5 -10)*5 for storage (up to 50 x) • (5 -10)*10 for CPU (up to 100 x) • At the same time, if we assume technology (“Moore and friends laws”) will improve at 20%/y, a factor 5 -6 is gained • RAW event size 1 5 MB/ev (factor 5 x) • We miss factors 5 -20 • Reconstruction times worse than linear. At least a factor 10 x slower • ”If we do nothing” not an option

AND EVEN THIS WOULD BE OPTIMISTIC

SOLUTIONS? • Please add money to HEP Computing • Today INFN funds resources for O(2 MEur) @ T 2 level and O(2 MEur) @ T 1 level (personnel and maintenance / operations costs NOT included) • A 5 x means (as a rule of thumb) that all CSN 1 budget is needed • Hope for technological breakthroughs • A leap in Moore’s law? Quantum Computing? Photonic CPUs? • Do less physics (for example, reconstruct electrons with p_T > 5 Ge. V only) • Severely limits the discovery potential of the HL-LHC experiments • Invest in long term R&D • can be top down or bottom up • Idea: do a first bottom up attempt; gather ideas (even absurd ones). At the same time make more people / FA / Experiments aware of the problem. Later collapse into something with a little more top down approach • HSF is the bottom up part of the effort

HSF • Initiated in 2014 • Activity level raised in 2017 • 2 large WSs • Preparation of the Community White Paper • Bottom up ideas in a range of Work Packages • Signed by “individuals” (no FA commitments, no experiment commitments) • A roadmap condensed document in ar. Xiv

STATUS FROM THE EXPERIMENTS • Of course, the experiments (*) were not sitting and waiting • Internal optimizations, reduction of storage copies, reduction of MC production lead to reductions year-to-year • 2017 Numbers from CMS and ATLAS speak about a factor 20 x wrt to 2017 resources • Which becomes an excess of a factor ~5 x when factoring Moore and friends • This is the size of the problem “today”, and what we should “earn back” • *: it is mostly ATLAS and CMS, since ALICE and LHCb live in a different LHC timescale and will have the major upgrade in 2021 • They are basically already done with modelling and R&D CMS Preliminary

THE WLCG STRATEGY DOCUMENT • If you want, the start of the top down process: WLCG taking note of CWP and Experiments starting to plan on it • The WLCG strategy document is a specific view of the CWP, prioritizing R&Ds relevant to the HL-LHC computing challenge • It is signed by Experiments: , it will get a review by LHCC and eventually an approval • It is the first step towards a TDR in 2022 (new!)

WHAT IS INSIDE? SOME GENERAL POINTS / TRENDS • Review the split between online and offline • ALICE already trying a merge; even if not at this level go to virtualized systems which can me moved to online or offline worlds when needed • Can we simulate the HL-LHC computing in such a way we can understand the effect on total cost by adjusting parameters? • Example: reduce the disk by 30% by doubling the WAN traffic? Is it worth? • Tune the data model, be able to find a small format which covers most of the analyses • Expand to expensive HPC systems for peak versus buy more CPUs • Use Virtual data: reproduce on the fly instead of storing (so store the recipes…) • Heterogeneous computing: on the fly discovery and utilization of local accelerators / GPU / FPGA • w/o code duplication? Open. CL? Client-server mode? • What is the cost of an inefficient use of HPC? • • Is ML going to be used in 1 -10 -50 -90% of the algorithms? • Will a 2026 standard machine _always_ have a GPU? • And the big one: • How many centers? Where to put storage? Where to put CPUs?

DISTRIBUTED COMPUTING – SOME GENERAL IDEAS • Data is the most precious asset from LHC: • Stays in our owned centers • CPU is almost stateless (at the level of few hours) • Can be anywhere • Provided they can access data (remote access) • Provided we can efficiently send data (network statinc/on demand) • Again, data lake in the next talk. Here the implications for anything outside the lake 1. Sites outside the lake can have storage, but it would be not centrally managed; most probably 1. A place to store your ntuples 2. A place to store your private MC productions 3. Simply a cache to make remote access faster 2. Sites can be storageless • This is at the hearth of the data lake model (see talk by D. Cesini) … but there is more • Keep our data safe • Data handling to become an infrastructure problem, not to be handled by experiments 1. An HPC center 2. An online farm by another experiment 3. Network is of capital importance here 1. Internal to the lake but also outside 2. Space / need for SDNs or similar: “next month I got an Amazon Grant, I need good connection to my site”

DATA AND COMPUTE INFRASTRUCTURES High Level Services Data (Lake) Infrastructure Storage Interoperability Services Distributed Regional Storage Asynchronous Data Transfer Distributed Storage Volatile Storage Compute Infrastructure Content Delivering and Caching Grid Compute Cloud Compute HPC Compute @HOME Grid Compute Provisioning Site N Simone. Campana@cern. ch - WLCG Overview Board 25/05/2018 12

POSSIBLE DISTRIBUTED COMPUTING CONFIGURATIONS • One: • A singe data lake roughly equivalent to today’s T 1 s, connected at least at 100 Gbit/s • T 1 s can host CPUs, or not • Experiments see a single storage entry point wrt to today’s 200 (life-is-good) • T 2 s become compute nodes (outside the lake, storage only as cache) • HPC centers as compute nodes • Commercial Clouds as Compute nodes • Two: • The data lake is composed of logical, not physical nodes: the total storage of Italian Computing centers is seen behind a single entry point • All T 1+T 2 s become part of the logical center which joins the lake • HPC centers as compute nodes • Three: • No data lake node in Italy (? ) • All our centers are Compute nodes, but we can afford a large cache in front of them, so that most of the accesses are local • But there is no custodiality commitment • Four: • US do not want to join a EU/CERN driven data lake (probable) • We have 2 or even more lakes, which have to have peerings between them • This makes Experiments life become complex: they see >1 storage endpoints, but still much less than 200

THE IMPORTANCE OF SOFTWARE • General feeling is that you will not able to solve the HL-LHC problems with computing itself • Data lake(s) can reduce the # of copies of a given file thanks to remote access • But not below 1 copy (and we are not far even now) • CPU cycles can be easier to obtain by allowing for HPC / Commercial / Diskless sites, but the total # of cycles is still needed • What can help is the ability to use new architectures? • 2000 -now: • HEP computing using the most performant hardware @ a given price: commodity machines are X 86 + Linux, exactly what we use • Already now-foreseeable future: • Best cycles/EUR are GPUs, FPGA, even ASIC • Can we use them? • How? • At which (human) cost? • Can we trust something (compilers, etc) will make them easily usable? (Open. CL, …)

A PROBABLE FUTURE … • Technology is now driven by Commercial Clouds • By far the highest buyers in the market • Eventually by HPC market (mainly for showcasing superior technology) • In both cases, the easiest to predict future is an ubiquitous presence of GPUs, Why? • Commercial: ML • HPC: best Linpack performers • This means most of the Flops in a chip will come from vector operations • Will be the cheapest / $ • Three ways to use them: 1. Specialized code: can be done for a small, mission critical part of the software (tracking, …) 2. Use Open. CL or friends to get “at least some benefits” 3. (use your faith) Use Machine Learning whenever possible • GPUs are seen via Tensor. Flow and not from the code we need to program • A major experiment starting with A is already claiming all the reconstruction will be ML based

CONCLUSIONS • The process to understand / simulate / plan for 2026+ HL-LHC computing has started • On paper, we are short by a factor O(5 x) • On the positive side, the 5 x factor means everyone goes on vacation for 10 years – not happening! • Ideas on how to reduce costs are available • Use of architectures cheaper (per flops) than x 86_64 • Reduce storage costs by federating storage areas, also limiting duplication • Allow the use of “unconventional” resources, like HPC, Disk less, Commercial • It is clear these are all at the level of ideas • R&Ds are of capital importance • Test beds will need to follow • New level of competences needed • How?