CMS Software Architecture Software framework services and persistency

CMS Software Architecture Software framework, services and persistency in high level trigger, reconstruction and analysis Real-Time Requirements and Implications for the FU Vincenzo Innocente CERN/EP/CMC 1/29/2022

CMS (analysis&reconstruction) Software Environmental data Request part of event Detector Control Quasi-online Reconstruction Online Monitoring store Event Filter Object Formatter Request part of event Store rec-Obj Request part of event store Persistent Object Store Manager Object Database Management System store Simulation G 3 and or G 4 Software Architecture and Data Model Vincenzo Innocente, CERN/EP Store rec-Obj and calibrations Data Quality Calibrations Group Analysis Request part of event User Analysis on demand

Use Cases (single event processing) (current functionality in ORCA) Simulated Hits Formatting Digitization of Piled-up Events DAQ & Online monitoring (today for Test-Beams) L 1 Trigger Simulation Track Reconstruction Calorimeter Reconstruction Global Reconstruction Physics Analysis Histogramming, Event visualization Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Use Cases (Job Level) (current functionality in ORCA) Select input event-collection Select events from collection Produce analysis-objects (data card or env variable) (code, or at link time) Choose to make persistent newly created objects Select output event-collection Select a physical-clustering strategy (location of output data) Select a detector and an algorithm configuration Decide if persistent objects are obsolete (and re-construct them) Perform shallow or deep copy of selected events Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Requirements (from the CTP) u. Multiple Environments: u Various software modules must be able to run in a variety of environments from level 3 triggering, to individual analysis u. Migration between environments: u Physics modules should move easily from one environment to another (from individual analysis to level 3 triggering) u. Migration u Should to new technologies: not affect physics software module Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Requirements (from the CTP) u. Dispersed code development: u The software will be developed by organizationally and geographically dispersed groups of part-time non-professional programmers u. Flexibility: u Not all software requirements will be fully known in advance Not only performance Also modularity, flexibility, maintainability, quality assurance and documentation. Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Strategic Choices Modular Architecture (flexible and safe) u Object-Oriented Framework u Strongly-typed interface Uniform and coherent software solutions u One main programming language u One main persistent object manager u One main operating system Adopt standards u Unix, C++, ODMG, Open. GL. . . Use widely spread, well supported products (with healthy future) u Linux, C++, Objectivity, Qt. . . Mitigate risks u Proper planning with milestones u Track technology evolution; investigate and prototype alternatives u Verify and validate migration paths; have a fall-back solution ready Software Architecture and Data Model Vincenzo Innocente, CERN/EP

CARF CMS Analysis & Reconstruction Framework Physics modules Specific Framework Event Filter Generic Application Framework Reconstruction Algorithms Calibration Objects Physics Analysis Configuration Objects Data Monitoring Event Objects CMS adapters and extensions Utility Toolkit ODBMS Software Architecture and Data Model Vincenzo Innocente, CERN/EP Geant 4 CLHEP Paw Replacement C++ standard library Extension toolkit

Reconstruction Scenario Reproduce Detector Status at the moment of the interaction: u front-end electronics signals (digis) u calibrations u alignments Perform local reconstruction as a continuation of the front-end data reduction until objects detachable from the detectors are obtained Use these objects to perform global reconstruction and physics analysis of the Event Store & Retrieve results of computing intensive processes Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Reconstruction Model Sim Hits Raw Data Detector Element Digis Rec Hits Geometry Conditions Algorithm Rec Objs Algorithm Software Architecture and Data Model Vincenzo Innocente, CERN/EP Rec Objs Event

Action on Demand Compare the results of two different track reconstruction algorithms Reco Hits Detector Element Hits Reco T 1 Event T 1 Calo. Cl T 2 Software Architecture and Data Model Vincenzo Innocente, CERN/EP Reco Calo. Cl Reco T 2 Analysis

Action on Demand in HLT Introducing and removing algorithms (even at run time) is straightforward and efficient u. A new filter algorithm will automatically trigger the creation and/or the loading of the objects it needs u Removing (or not running) a filter will automatically inhibit the creation of the objects it depends upon (including geometry, conditions, algorithms etc. ) HLT can work on best effort: u fully reconstruct and fully store events for express-line analysis u partially reconstruct and store just raw data for the rest u read and store partial events for calibration purposes u Offline jobs will identify what has been already reconstructed, what is obsolete, what is missing, and bring the analysis to completion with no user intervention Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Event Builder/Filter (Real World Object Model) BU Read Out Unit FED 1 FU 1. . m Event Software Architecture and Data Model Vincenzo Innocente, CERN/EP FED, ROU, BU, FU, all assemble event fragments from data coming higher in the chain. Current wisdom dictates that BU assembles full events, ship fragments to FU and keeps an event until FU decides its fate. The whole design should accommodate technology evolution that could modify protocols, buffer management and network connections at any level

FU “input” software (evolution of Test-Beam prototypes) u. BU Proxy u Handle connection with real BU (or any other input device) u Unpack received fragments u Push data into Read Out Units u. Read Out Unit u Receive raw-data from BU Proxy u Push FED data in the corresponding Detector Element èan intermediate FED object may be required u Perform AS-IS raw-data persistent storage u. Detector Element u Request Data from Read Out Unit u Decode FED data u Cache decoded and reconstructed data. u Supply data to reconstruction algorithms and to persistent storage manager Software Architecture and Data Model Vincenzo Innocente, CERN/EP

FU “output” FU should be capable of persistently store any raw and reconstructed data corresponding to accepted events. Which objects to store and how to clusterize them should be configurable. All this is common to all reconstruction processes. Current solution is that the FU formats the data in its final storage representation and write them to a device that, at configuration time, has been chosen to be a local disk or a remote data server. If raw-data has to be stored, the safest and more flexible solution is to save them in their original format at Read Out Unit level granularity. Software Architecture and Data Model Vincenzo Innocente, CERN/EP

CMS Test Beam setup I/O performance Test April 1999 (published in RD 45 LHCC report) 1/29/2022

Hardware Configuration u System: Sun. Enterprise 450 (4 X Ultr. SPARC-II 296 MHz) u Memory size: 512 Megabytes u Page Size: 8 KB u Disk Controllers: 2 X Dual FW-SCSI-2 (Symbios, 53 C 875) u Disks: 20 X FUJITSU MAB 3091 S SUN 9. 0 G Rev. 1705 OS: Solaris 5. 6 File-system Configuration u 80 GB striped file-system of 19 slices 128 K "away" Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Simple “C++” I/O Writing 2 GB files in “records” of 8 KB using ofstream: : write (egcs 1. 1) 5 processes 90 MB/s Less than 50% cpu load Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Objectivity Test “H 2” like raw data structure u random size objects (average 2. 4 KB useful data) u 4000 events with 45 raw-objects each u 180 K raw-objects in total corresponding to 436 MB of useful data u including event structure and overhead a total of 469 MB written to disk per “run” Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Objectivity optimization 32 K page size INDEX mode 0 100 page cache size no associations (only VArray. T<oo. Ref()>) commit and hold each 1000 events Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Results (local database) 12 processes 22 MB/s 100% cpu usage! Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Distributed federation Remote lock server uless than 10% degradation Remote “System Databases” uinitialization èfrom time increases by a factor 5 to 10 few seconds to several tens udatabase population: less than 10% degradation Remote Event database ulimitation from present network speed (1 MB/s? ? ) Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Conclusions from I/O test (1) Using a “state of the art” commercial disk server we can write at a speed of 90 MB/s using a simple C++ program On the same system, writing a more realistic rawdata event structure with Objectivity/DB, we can reach 22 MB/s Running on two such systems I reached a peak “sustained” aggregate speed of ~40 MB/s Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Conclusion from I/O test (2) The CMS online farm is foreseen to have a power of 4. 5 Tips distributed over 1000 cpus Today, to write with Objectivity at 100 MB/s, a power of about 6 Gips, distributed over 20 cpus, is required We can safely estimate that “Objectivity formatting” of raw data will require less than 1% of the CMS online computing power Software Architecture and Data Model Vincenzo Innocente, CERN/EP

FU “output”: alternative solutions One alternative is to stream full reconstructed events, using some intermediate proprietary format, to a buffer-system and format them asynchronously. u If event formatting is required, it will require a similar amount of memory & cpu u It could be faster if large-buffer sequential-streaming is used The fall back solution is to stream raw-data as a single complete event record in native format directly from the BU (or the BUProxy) to the fastest and closest device and perform any additional formatting offline. The final bottleneck will most probably be the network Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Conclusions Current CMS software architecture was designed with HLT requirements in mind: u Configurable framework u Action on demand u Plug-in physics modules Several Filter Farm peculiarities already prototyped in Test-Beam The use of uniform and coherent software solutions will make online-offline software migration possible Risks can be mitigated u Using the framework to shield physics modules from the underlying technology (will not penalize performance) u Having fall-back solutions ready Software Architecture and Data Model Vincenzo Innocente, CERN/EP

A Uniform and Coherence Approach to Object Persistency Vincenzo Innocente CERN/EP/CMC 1/29/2022

HEP Data User Tag (N-tuple) Environmental data Detector and Accelerator status u Calibrations, Alignments u Tracker Alignment Event-Collection Data (luminosity, selection criteria, …) … Event Data, User Data Collection Data Tracks Event Collection Electrons Event Navigation is essential for an effective physics analysis Complexity requires coherent access mechanisms Software Architecture and Data Model Vincenzo Innocente, CERN/EP Ecal calibration

Uniform approach Coherent data access model Save effort Leverage experience Reuse design and code Main road in producing better and higher quality software Software Architecture and Data Model Vincenzo Innocente, CERN/EP

CMS needs a serious DBMS An experiment lasting 20 years can not rely just on ASCII files and file systems for its production bookkeeping, “condition” database, etc. Even today at LEP, the management of all real and simulated data-sets (from raw-data to n-tuples) is a major enterprise u Multiple models used (DST, N-tuple, HEPDB, FATMAN, ASCII) A DBMS is the modern answer to such a problem and, given the choice of OO technology for the CMS software, an ODBMS (or a DBMS with an OO interface) is the natural solution for a coherent and scalable approach. Software Architecture and Data Model Vincenzo Innocente, CERN/EP

CMS Experience Designing and implementing persistent classes not harder than doing it for native C++ classes. Easy and transparent distinction between logical associations and physical clustering. Fully transparent I/O with performances essentially limited by the disk speed (random access). File size overhead (5% for realistic CMS object sizes) not larger than for other “products” such as ZEBRA, BOS etc. Objectivity/DB (compared to other products we are used to) is robust, stable and well documented. It provides also many additional useful features. All our tests show that Objectivity/DB can satisfy CMS requirements in terms of performance, scalability and flexibility Software Architecture and Data Model Vincenzo Innocente, CERN/EP

CMS Experience There additional “configuration elements” to care about: ddl files, schema-definition databases, database catalogs u organized software development: rapid prototyping is not impossible, its integration in a product should be done with care Performance degradations often wait you around the corner u monitoring of running applications is essential, off-the-shell solutions often exist (Ba. Bar, Compass, CMS) Objectivity/DB is a “bare” product: u integration into a framework is our responsibility Objectivity is slow to apply OUR changes to their product u Is this a real problem? Do we really want a product whose kernel is changed at each user request? Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Alternatives: ODBMS Versant is a viable commercial alternative to Objectivity u do we have time to build an effective partnership (eg. MSS interface)? Espresso (by IT/DB) should be able to produce a fully fledged ODBMS in a couple of years once the proof-of-concept prototype is ready IT restructuring and Hoffman review have delayed Espresso We hope to be able to resume Espresso tests soon Migrate CARF from Objectivity to another ODBMS u We expect that it would take about one year u Such a transition will not affect the basic principles of CMS software architecture and Data Model and u Will involve only the core CARF development team. u Will not disrupt production and physics analysis Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Alternatives: ORDBMS (Relational DB with OO interface) are appearing on the market Up to now they looked targeted to those who have already a relational system and wish to make a transition to OO A New ORACLE product has the appearance of a fully fledged ODBMS IT/DB is in the process of evaluating this new product as an event store If these early prototypes will look promising CMS will join this evaluation next year. We hope to be able to assess the impact of ORDBMS on CMS Data Model and on migration effort before the end of 2001 Software Architecture and Data Model Vincenzo Innocente, CERN/EP

Fallback Solution: Hybrid Models u (R)DBMS for Event Catalog, Calibration, etc u Object-Stream files for event data u Ad-hoc networked dataserver and MSS interface Less flexible u Rigid split between DBMS and event data u One way navigation from DBMS to event data More complex u Two different I/O systems u More effort to learn and maintain This approach will be used by several experiment at BNL and Fermi. Lab u (RDBMS not directly accessible from user applications) CMS and IT/DB are following closely these experiences. We believe that this solution could seriously compromise our ability to perform our physics program competitively Software Architecture and Data Model Vincenzo Innocente, CERN/EP

ODBMS Summary A DBMS is required to manage the large data set of CMS (including user data) An ODBMS provides a coherent and scalable solution for managing data in an OO software environment Once an ODBMS will be deployed to manage the experiment data, it will be very natural to use it to manage any kind of data related to detector studies and physics analysis Objectivity/DB is a robust and stable kernel ideal to be used as the base to build a custom storage framework Realistic alternatives start to appear on the market: CMS actively follows these developments Software Architecture and Data Model Vincenzo Innocente, CERN/EP