The Performance and Scalability of the backend DAQ

The Performance and Scalability of the back-end DAQ sub-system Igor SOLOVIEV CERN ATLAS DAQ/EF-1 9/18/2020 CHEP 2000

Contents l Introduction l l l Test Results l l l ATLAS DAQ/EF P-1 project back-end software overview & architecture component test results integrated back-end sub-system test results Summary and Future 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ 2

ATLAS DAQ/EF P-1 Project l Goal: to produce a prototype system representing a “full slice” of a DAQ suitable for evaluating candidate technologies and architectures for the final ATLAS DAQ system l Sub-systems: l l l Detector Interface Data-Flow Event Filter Back-end Status l l l The Perfor mance and Scalab ility of the backend DAQ Base-line system developed & working in lab. environment Exploitation phase up to TDR (2001) To be used on test-beam (summer 2000) 9/18/2020 3

Back-end Sub-system Is used to configure, control and monitor the DAQ system l It excludes management, processing and transportation of physical data l It talks to all the other online systems (“glue” of the experiment) l More information: l l l 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ WWW pages: http: //atddoc. cern. ch/Atlas/ “Impact of Software Review and Inspection”: talk F 331, today 17: 50, Doris Burckhart 4

Back-end Architecture l Components l l Operational environment: l l l split back-end software into groups with similar functionality (Core + TDAQ & detector integration components) heterogeneous collection of UNIX workstations, PCs and embedded systems (e. g. PPC on VME under realtime Lynx OS) connected via a local network developed in C++ and ported to several compilers on Solaris, Linux, Lynx, HP-UX & Window NT The Design: l 9/18/2020 Perfor mance and Scalab ility of the backend DAQ use freeware and commercial software: Tools. h++, OODB, CORBA, CHSM, CLIPS, Motif/Java 5

Back-end core components l Configuration Databases describes all aspects of the configuration l Information Service (IS) general purpose information exchange facility l Message Reporting System (MRS) allows software components to report messages in distributed environment l Process Manager (PMG) The Perfor mance and Scalab ility of the backend DAQ performs distributed job control of components l Run Control (RC) controls configuration and data taking operations 9/18/2020 6

Component Unit Tests Results l Configuration Databases l l used by many components during system start-up tests done for different OKS configurations (single readout crate, typical P-1 conf. , expected ATLAS DAQ conf. ) on average workstation time to load P-1 conf. , make complete traverse and close is about 1. 5 sec. and on PPC VME board the same test requires about 3 sec. Information Systems (IS & MRS) l l l 9/18/2020 used by many components during all phases of system operation (publish/subscribe facilities) scalable (multiple servers to split the load) The Perfor benchmarks done on single workstation and on several mance computers for different conf. (size, up to 50+10 clients) and Scalab the response time is a few milliseconds ility of the better results for distributed systems backend DAQ 7

Component Unit Tests Results l Process Manager l l used during the system start-up and shutdown results obtained on single Solaris workstation time to start a process is a few 100 s milliseconds and slowly increases with the number of managed processes Run Control l l 9/18/2020 required to change the state of the system scalable by changing the structure of RC tree tests on all available workstations (up to 250 controllers) The to change the system’s state with several 10 s of nodes Perfor varies from several 100 s milliseconds up to few seconds mance and depending on the state of the system Scalab the time to change running/configured states is <1 ilitysec. of the backend DAQ 8

Component Unit Tests Conclusions l Unit tests made for back-end core components show that they are in accordance with DAQ P-1 requirements l Similar tests will be done for back-end integration components 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ 9

Back-end Sub-system Tests l What: l l Why: l l bring together all the core and several TDAQ/detector integration components to simulate the control & configuration of data taking sessions Where: l l 9/18/2020 back-end servers are running on UNIX workstation The Perfor others (PMG agent, LDAQ emulator & RC Ctrl. ) on PC mance running Linux or VME based Power PC CPU board and Scalab running Lynx OS ility of the backend DAQ 10

Test Configurations Network PMG Agent G IPC P IPC DF IS PMG IS RDB RM MRS DAQ Supervisor RC Root Ctrl 9/18/2020 RC IS MRS L IGUI PMG Agent LDAQ RC Ctrl PMG Agent The Perfor LDAQ mance and RC Ctrl Scalab ility of the backend DAQ 11

Test Description Done by shell script: l l l start communication services launch configuration processes via DAQ supervisor marshal the hierarchy of RC controllers through different states: I - L - C - R - C - L - I stop DAQ supervisor processes R R B - booted stop servers C L B setup 9/18/2020 I cold start C warm stop warm start l I - initialized L - loaded C - configured R - running The C L I cold stop B Perfor mance and Scalab ility of the backend DAQ shutdown 12

Time (seconds) Start-up & warm start/stop Number of processors/crates Power. PC 100/200 MHz 32/64 MB Lynx OS 9/18/2020 The Perfor mance Number of processors/crates and Pentium III 450 MHz 128 MB Scalab ility of Linux the backend DAQ 13

Time (seconds) Start-up & close Number of processors/crates Power. PC 100/200 MHz 32/64 MB Lynx OS 9/18/2020 The Perfor mance Number of processors/crates and Pentium III 450 MHz 128 MB Scalab ility of Linux the backend DAQ 14

Back-end system test summary l Results l l l time to start/stop processes depends on OS, computer architecture and configuration once all processes started, the time to change system state remains constant (good distributed control) the use of IS, MRS and conf. db has a negligible effect on the performance the results even for the largest configurations is in acceptable range (< 1 minute to start-up on Linux) Known problems l l 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ pmg agents started via RSH with long delays (20 sec) the computers were not dedicated to tests 15

Summary & Future l Individual back-end component test l l l Integrated back-end system tests l l done for core components and show that they are in accordance with the DAQ/EF P-1 requirements similar tests have to be done for integration components performed employing the majority of the components verified correct component inter-operation, ability to work in a distributed multi-platform environment gathered performance measurements Future l l 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ more statistics for larger configurations (more hosts) script improvement and better start-up/shutdown synchronization 16

Appendix: Configuration Databases l Importance l l Results (with OKS) Time (s) l are used by many components during initialization performance is important for system start-up 1 single read-out crate The 10 prototype -1 Perfor 200 expected ATLAS DAQ mance Number of crates 9/18/2020 and Scalab ility of the backend DAQ 17

Appendix: Information Service Importance l l l used by many components performance is important during all phases of system operation Results l l l scalable (multiple servers to split the load) update medium size info. results presented (on single host) similar to publish and remove Update time (ms) l Number of sources 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ 18

Appendix: Message Reporting System Importance l l l used by many components performance is important during all phases of system operation Results l l 9/18/2020 presented tests obtained on single host better results obtained in distributed environment Report time per message (ms) l Number of senders The Perfor mance and Scalab ility of the backend DAQ 19

Appendix: Process Manager l Importance performance is important for system start-up and shutdown Results obtained on single Solaris workstation 9/18/2020 Time per process (ms) l The Perfor mance and Scalab ility of the backend DAQ 20

Appendix: Run Control l Importance required to change the state of the system Results l l scalable by changing the structure of the RC tree tests done on all available workstations Time, (s) l Number of controllers 9/18/2020 The Perfor mance and Scalab ility of the backend DAQ 21