CMS Software Architecture An experience in OO C

CMS Software Architecture An experience in OO & C++ Vincenzo Innocente CERN 1/3/2022

CMS (offline) Software Environmental data Request part of event Slow Control Quasi-online Reconstruction Online Monitoring store Event Filter Objectivity Formatter Request part of event Store rec-Obj Request part of event store Persistent Object Store Manager Object Database Management System Store rec-Obj and calibrations store Simulation G 3 and or G 4 1/3/2022 Data Quality Calibrations Group Analysis Vincenzo Innocente CMS Architecture Request part of event User Analysis on demand 2

Requirements (from the CTP, dec. `96) u. Multiple Environments: u Various software modules must be able to run in a variety of environments from level 3 triggering, to individual analysis u. Migration between environments: u Physics modules should move easily from one environment to another (from individual analysis to level 3 triggering) u. Migration u Should 1/3/2022 to new technologies: not affect physics software module Vincenzo Innocente CMS Architecture 3

Requirements (from the CTP) u. Dispersed code development: u The software will be developed by organizationally and geographically dispersed groups of part-time non-professional programmers u. Flexibility: u Not all software requirements will be fully known in advance Not only performance Also modularity, flexibility, maintainability, quality assurance and documentation. 1/3/2022 Vincenzo Innocente CMS Architecture 4

Technologies Do not always jump on next year buzzword Do not limit to technologies standardized in times when graduate students were not born yet There is no Silver Bullet Any single technical issue can be solved with few thousand lines of code by any of us. This is not the point: What is needed is a coherent Software Architecture for an experiment which will last longer than a decade 1/3/2022 Vincenzo Innocente CMS Architecture 5

C++ is a very advanced language which supports u. C style coding (for algorithms it is ok) è but u Object it is the source of all evils (as memory leaks) Oriented programming è Data Hiding è Encapsulation è Polymorphism è Multiple and Virtual inheritance u Generic Programming è Templates è Parametric u. All 1/3/2022 Polymorphism this makes it complex but powerful! Vincenzo Innocente CMS Architecture 6

Use Cases Simulated Hits Formatting Digitization of Piled-up Events Test-Beam DAQ & Analysis L 1 Trigger Simulation Track Reconstruction Calorimeter Reconstruction Global Reconstruction Physics Analysis 1/3/2022 Vincenzo Innocente CMS Architecture 7

Track Reconstruction Local measurements belongs to detector element and are affected by the detector element state (calibrations, alignments) Pattern recognition navigates in the detector to associate local measurements into a track 1/3/2022 Vincenzo Innocente CMS Architecture 8

Global Reconstruction Global reconstruction is performed in an absolute reference frame 4 -vector-like objects are built out of trajectories and localized energy deposits A wide range of particle identification, jet, vertex, etc algorithms can be applied to produce others 4 -vector -like objects Access to the original detector data maybe required 1/3/2022 Vincenzo Innocente CMS Architecture 9

Reconstruction Scenario Reproduce Detector Status at the moment of the interaction: ufront-end electronics signals (digis) ucalibrations ualignments Perform local reconstruction as a continuation of the front-end data reduction until objects detachable from the detectors are obtained Use these objects to perform global reconstruction and physics analysis of the Event Store & Retrieve results of computing intensive processes 1/3/2022 Vincenzo Innocente CMS Architecture 10

Reconstruction Sources 1/3/2022 Vincenzo Innocente CMS Architecture 11

Components Reconstruction Algorithms Event Objects Physics Analysis modules Other services (detector objects, environmental data, parameters, etc) Legacy not-OO data (GEANT 3) The instances of these components require to be properly orchestrated to produce the results as specified by the user 1/3/2022 Vincenzo Innocente CMS Architecture 12

CARF CMS Analysis & Reconstruction Framework Physics modules Application Framework Reconstruction Algorithms Event Filter Physics Analysis Calibration Objects Data Monitoring Visualization Objects Event Objects Utility Toolkit LHC++ ODBMS 1/3/2022 Geant 4 CLHEP Vincenzo Innocente CMS Architecture PAW Successor C++ standard library Extension toolkit 13

Architecture structure An application framework CARF (CMS Analysis & Reconstruction Framework), customisable for each of the computing environments Physics software modules with clearly defined interfaces that can be plugged into the framework A service and utility Toolkit that can be used by any of the physics modules Nothing terribly new, but. . . Traditional architecture can not cope with LHC-collaboration complexity 1/3/2022 Vincenzo Innocente CMS Architecture 14

Problems with traditional architectures Traditional Framework schedules a-priori the sequence of operations required to bring a given task to completion Major management problems are produced by changes in the dependencies among the various operations u. Example 1: Reconstruction of track type T 1 requires only tracker hits Reconstruction of track type T 2 use calorimetric clusters as seeds If a user switches from T 1 to T 2 the framework should determine that calorimeter reconstruction should run first now u. Example 2: The global initialization sequence should be changed because, for one detector, some condition changes often than foreseen 1/3/2022 Vincenzo Innocente CMS Architecture 15

Framework Basic Dynamics Avoid monolithic structure Collection of loosely coupled mechanisms which implements in abstract the tasks of a HEP reconstruction and analysis software. Implicit Invocation Architecture No central ordering of actions, no explicit control of data flow: only implicit dependencies External dependencies managed through an Event Driven Notification to subscribers Internal dependencies through an Action on Demand mechanism 1/3/2022 Vincenzo Innocente CMS Architecture 16

Event Driven Notification Observers are instantiated by static factories residing in shared libraries. These are loaded on demand during Dispatcher application configuration Detector elements observe physics events Factories observe user requests Obs 1 1/3/2022 Obs 3 Vincenzo Innocente CMS Architecture Obs 4 Observers clients or providers 17

Action on Demand Compare the results of two different track reconstruction algorithms Rec Hits Detector Element Hits Event Rec T 1 Calo. Cl T 2 1/3/2022 Rec Calo. Cl Rec T 2 Vincenzo Innocente CMS Architecture Analysis 18

HEP Data User Tag (N-tuple) Environmental data u Detector and Accelerator status u Calibrations, Alignments Event-Collection Meta-Data (luminosity, selection criteria, …) Tracker Alignment … Event Data, User Data Tracks Collection Meta-Data Event Collection Ecal calibration Electrons Event 1/3/2022 Vincenzo Innocente CMS Architecture 19

Do I need a DBMS? (a self-assessment) Do I encode meta-data (run number, version id) in file names? How many files and logbooks I should consult to determine the luminosity corresponding to a histogram? How easily I can determine if two events have been reconstructed with the same version of a program and using the same calibrations? How many lines of code I should write and which fraction of data I should read to select all events with two ’s with p > 11. 5 Ge. V and | |<2. 7? The same at generator level? If the answers scare you, you need a DBMS! 1/3/2022 Vincenzo Innocente CMS Architecture 20

A major challenge for LHC: The scale Event output rate 100 events/sec (109 events/year) Data written to tape 100 M Bytes/sec (1 PB/yr) Processing capacity > 10 TIPS (= 1013 instr. /s) Typical networks Hundreds of Mbits/second Lifetime of experiment 2 -3 decades Users ~1700 physicists Software developers ~100 è~100 Petabytes Total for the LHC 1/3/2022 Vincenzo Innocente CMS Architecture 21

Can CMS do without a DBMS? An experiment lasting 20 years can not rely just on ASCII files and file systems for its production bookkeeping, “condition” database, etc. Even today at LEP, the management of all real and simulated data-sets (from raw-data to n-tuples) is a major enterprise. A DBMS is the modern answer to such a problem and, given the choice of OO technology for the CMS software, an ODBMS (or a DBMS with an OO interface) is the natural solution. 1/3/2022 Vincenzo Innocente CMS Architecture 22

A “BLOB” Model Event Data. Base Objects Raw. Event Rec. Even t Blob: Blob a sequence of bytes. Decoding it is a “user” responsibility. Blob 1/3/2022 Why should Blobs not be stored in the DBMS? Vincenzo Innocente CMS Architecture Blob 23

Raw Event Index Raw. Event Raw. Data belonging to different “detectors” are clustered into different containers. The granularity will be adjusted to optimize I/O performances. Read. Out Raw. Dat a Vector of Digi . . . Raw. Dat a An index at Raw. Event level is used to avoid the access to all containers in search for a given Raw. Data. Vector of Digi Index implemented as an ordered vector of pairs 1/3/2022 Raw. Data are identified by the corresponding Read. Out. Vincenzo Innocente CMS Architecture A range index at Raw. Data level could be used for fast random access in complex detectors. 24

A major challenge for LHC: The scale Event output rate 100 events/sec (109 events/year) Data written to tape 100 M Bytes/sec (1 PB/yr) Processing capacity > 10 TIPS (= 1013 instr. /s) Typical networks Hundreds of Mbits/second Lifetime of experiment 2 -3 decades Users ~1700 physicists Software developers ~100 è~100 Petabytes Total for the LHC 1/3/2022 Vincenzo Innocente CMS Architecture 25

Can CMS do without a DBMS? An experiment lasting 20 years can not rely just on ASCII files and file systems for its production bookkeeping, “condition” database, etc. Even today at LEP, the management of all real and simulated data-sets (from raw-data to n-tuples) is a major enterprise. A DBMS is the modern answer to such a problem and, given the choice of OO technology for the CMS software, an ODBMS (or a DBMS with an OO interface) is the natural solution. 1/3/2022 Vincenzo Innocente CMS Architecture 26

Physical clustering 1/3/2022 Vincenzo Innocente CMS Architecture 27

Can every object have its own persistency? Data size Data complexity Self-Description: which granularity? Meta-Data vs Data logical vs physical organization Flexibility vs Efficiency Interface with “standard” tools (like GUIs) Fast prototyping vs formal/controlled design User knowledge and training 1/3/2022 Vincenzo Innocente CMS Architecture 28

Is an ODBMS an overkill for Histograms? Maybe, if histograms are your sole I/O. (I use my sun ultra-5 to read mails through pine even if a line-mode terminal would be more than adequate) N-tuples are “user” event-data and, for any serious use, require a level of management and book-keeping similar to the “experimentwide” event data. What counts is the efficiency and reliability of the analysis: The most sophisticated histogramming package is useless if you are unable to determine the luminosity corresponding to a given histogram! 1/3/2022 Vincenzo Innocente CMS Architecture 29

Objectivity Features CMS (really) uses Persistent objects are real C++ (and Java) objects I/O cache (memory) management u no explicit read and write u no need to delete previous event Smart-pointers (automatic id to pointer conversion) Efficient containers by value (VArray) VArray Full direct navigation in the complete federation Flexible object physical-clustering Object Naming u as top level entry point (at “collection” level) u as rapid prototyping tool 1/3/2022 Vincenzo Innocente CMS Architecture 30

Additional ODBMS (Objy) Advantages Novel access methods: u. A collection of “electrons” with no reference to events u Direct reference from event-objects to “condition database” u Direct reference to event-data from user-data Flexible run-time clustering of heterogeneous-type objects u cluster together all tracks or all objects belonging to the same event Real DB management of reconstructed objects u add 1/3/2022 or modify in place and on demand parts of an event Vincenzo Innocente CMS Architecture 31

CMS Experience (Pro) Designing and implementing persistent classes not harder than doing it for native C++ classes. Easy and transparent distinction between logical associations and physical clustering. Fully transparent I/O with performances essentially limited by the disk speed (random access). File size overhead (3% for realistic CMS object sizes) not larger than for other “products” such as ZEBRA or BOS. Objectivity/DB (compared to other products we are used to) is robust, stable and well documented. It provides also many additional useful features. All our tests show that Objectivity/DB can satisfy CMS requirements in terms of performance, scalability and flexibility 1/3/2022 Vincenzo Innocente CMS Architecture 32

CMS Experience (Cons) Objectivity (and the compilers it supports) does not implement the “latest” C++ features (changing: fast convergence toward ANSI standard) There additional “configuration elements” to care about: ddl files, schema-definition databases, database catalogs u organized software development: rapid prototyping is not impossible, its integration in a product should be done with care Performance degradations often wait you around the corner u monitoring of running applications is essential, off-the-shell solutions often exist (Ba. Bar, Compass) Objectivity/DB is a “bare” product: u integration into a framework is our responsibility Objectivity is slow to apply OUR changes to their product u Is this a real cons? Do you really want a product whose kernel is changed at each user request? 1/3/2022 Vincenzo Innocente CMS Architecture 33

CMS Experience (missing features) Scalability: 64 K files are not enough (Objy is working on it) containers are the natural Objectivity units, still things for which the OS (and files) is preferred u “bulk” data transfer (to mass-storage, among sites) u access control, space allocation to users, etc. Efficient and secure AMS (ok in 5. 2!!!) u with MSS and WAN support Activator & de-activator (part of ODMG standard) u to implement “transient parts” of persistent objects Support for “private” user classes and user data experiment-wide ones) (w. r. t. u many custom solution based on multi-federation u Active schema User Application Layer u like 1/3/2022 a rapid prototyping environment Vincenzo Innocente CMS Architecture 34

ODBMS Summary A DBMS is required to manage the large data set of CMS (including user data) An ODBMS is the natural choice if OO is used in all SW There is no reason NOT to store event-data in the DB as a “Blob” or as a real object system Once an ODBMS will be deployed to manage the experiment data, it will be very natural to use it to manage any kind of data related to detector studies and physics analysis Objectivity/DB is a robust and stable kernel ideal to be used as the base to build a custom storage framework 1/3/2022 Vincenzo Innocente CMS Architecture 35

Conclusions Object Oriented technologies have proven to be required to develop flexible software architectures C++ is the natural choice for a large project: today JAVA can be a realistic alternative OO and C++ as been easily adopted by all detector developers (see C. Grandi, T. Todorov & A. Vitelli CHEP talks) ODBMS is a robust technology ideal for building a large coherent object store CMS MC production 2000 will exercise on a real scale all this 1/3/2022 Vincenzo Innocente CMS Architecture 36

Object Model 1/3/2022 Vincenzo Innocente CMS Architecture 37