Distributed Control and Monitoring of HighLevel Trigger Processes
Distributed Control and Monitoring of High-Level Trigger Processes on the LHCb on-line Farm Philippe Vannerem CERN / EP ICALEPCS - Oct 03
Contents Ø the Experiment Control System of LHCb Ø the Event Filter Farm Ø GAUDI, the software framework for event processing Ø GAUCHO, how to glue GAUDI to SCADA Ø Communicating histograms Ø Conclusions ICALEPCS / Oct-03 Philippe Vannerem 2
ECS Scope Experiment Control System DCS Devices (HV, LV, GAS, Temperatures, etc. ) Detector Channels Trigger Front End Electronics Readout Network Processing/Filtering Storage DAQ External Systems (LHC, Technical Services, Safety, etc) ICALEPCS / Oct-03 Philippe Vannerem 3
ECS: Generic SW Architecture LHC GAS . . . Det. Dcs 1 Sub. Sys 1 DAQ DCS Det. Dcs. N Sub. Sys 2 Dev 1 Dev 2 Det. Daq 1 Sub. Sys. N Dev 3 DSS . . . Dev N To Devices (Hardware or Software) Ø Distributed Hierarchical Control Ø Commercial components and industrial standards ICALEPCS / Oct-03 Philippe Vannerem 4 Commands T. S. Status & Alarms ECS
ECS: Hierarchical control Ø Each node is able to control and monitor its subtrees: • Propagate commands to child level • Summarize information for the parent level • Implement specific behaviour & Take local decisions – Sequence & Automate operations – Recover errors • Partitioning – Exclude one or more of its subtrees which then can run stand-alone • User Interfacing – Present information and receive commands – Filter and display alarms ICALEPCS / Oct-03 Philippe Vannerem H V DCS Tracke r Tem p Muon H V GA S 5
ECS: implementation Ø Implementation in commercial SCADA system software: PVSS II • device description + access (OPC, Profibus, drivers) • alarm handling+display • user interface builder + control scripting • archiving Ø Joint COntrols Project between the LHC experiments: • The Framework: hierarchical control templates + tools and control modules for common HW in PVSS • Important building block: SMI++: – State Management Interface, distributed control – Abstract behaviour modeling Finite State Machines – Automation & Error recovery with a Rule based system Ø LHCb ECS is based the JCOP Framework and will extend it for its specific needs. ICALEPCS / Oct-03 Philippe Vannerem 6
LHCb’s DAQ&trigger architecture Level-1 Traffic 126 Links 1100 / 25 = 44 k. Hz 5. 5 GB/s Front-end Electronics FE FE FE TRM Switch HLT Traffic 323 Links 40 / 10 = 4 k. Hz 1. 6 GB/s Switch Multiplexing Layer 64 Links 32 Links L 1 -Decision Readout Network Storage System Gb Ethernet Level-1 Traffic HLT Traffic Mixed Traffic ICALEPCS / Oct-03 SFC Switch CPU CPU CPU … Sorter 94 Links 7. 1 GB/s SFC Switch CPU CPU CPU Philippe Vannerem TFC System 94 SFCs ~1800 CPUs CPU Farm 7
Event Filter Farm Control Ø EFF: ~1800 commodity rack- mounted CPUs Ø CPUs+SFCs linked to ECS PCs in separate Ethernet LAN Ø ECS Tasks: • Hardware control (remote boot, reset, …) + monitoring (CPU usage, memory, IO, etc. ) • Software control + monitoring Ø Level-1+ High-Level Trigger algorithms are implemented using the GAUDI framework ICALEPCS / Oct-03 Event Builder Switch SFC SFC CPU CPU . . . CPU Philippe Vannerem CPU . . . CPU 8
GAUDI Framework: design choices Ø Ø Separation between data and algorithms Separation between “transient” and “persistent” data Generic component interfaces OO Framework implemented with C++, runs on Win 2000+Linux ICALEPCS / Oct-03 Philippe Vannerem 9
GAUDI: component architecture Ø Interface defined in C++ pure virtual class Ø User specifies the components to be used in the Joboptions file Ø DLLs are loaded at run-time Ø Algorithms can be configured in the joboptions file ICALEPCS / Oct-03 Philippe Vannerem 10
Interface Requirements Ø Real-time system to use on the EFF Ø Interface to the trigger algorithms written in Gaudi event data processing framework. Ø Interface to the SCADA-based ECS implemented in PVSS. Ø Light-weight software: • Low CPU load to interfere as little as possible with event data crunching. • Disk-less operation on the nodes. • Low network traffic. Ø Flexible service-like component: easy to change/add monitored variables by physicist-Gaudi-users (trigger people). Ø Scalable with number of nodes Ø Compatible with farm partitioning. ICALEPCS / Oct-03 Philippe Vannerem 11
GAUCHO: GAUdi Component Helping Online Ø Gaucho package: monitor and control the Gaudi work on the Ø Ø farm. “users” (=future trigger algorithm programmers) can use this component in their Gaudi application to use the Gaudi-style Algorithm Service “Monitor. Svc” and then declare the variables to be published in their algorithms. “Monitor. Svc” publishes internal variables (integer, float, string) and histograms in the transient store using the Dim. Monitor. Server. A Dim. Property. Server gets/sets values of the properties of the Application. Mgr and Algorithms (configuration, steering). A Dim. Controller with internal states receives commands via DIM and steers the Application. Mgr. ICALEPCS / Oct-03 Philippe Vannerem 12
Gaudi-Gaucho-SCADA farm node - Gaudi DIM Application. Mgr monitor node - PVSS HLT Algorithm Dim. Controller DIM API Histogram. Svc Event. Data. Svc Detector. Data. Svc Monitor. Svc Gaucho Control. Mgr Dim. Property. Svr Control. Panel Dim. Monitor. Svr Monitor. Panel ICALEPCS / Oct-03 Philippe Vannerem 13
Inter-process tool: DIM: Ø Distributed Information Management system Ø Light-weight inter-process communication package based on the client/server paradigm Ø Asynchronous interrupt-driven data transfer across platforms Ø Here: C++ version used ICALEPCS / Oct-03 Philippe Vannerem 14
SCADA: controlling Gaudi apps Ø PVSS control panel for EFF. Ø Control. Mgr runs control script Ø Subfarm functionality: • Distributed control of its nodes Ø Farmnode functionality: • Connect to Dim. Controller, which steers Application. Mgr with commands (e. g. configure, start, pause, stop) • Mapping of finite state machine in Gaudi. Dim. Controller in PVSS FSM control object • Display tree of the configured sequence of Algorithms by talking to Dim. Property. Svr • Get/Set Properties of Algorithms ICALEPCS / Oct-03 Philippe Vannerem 15
SCADA: monitoring Gaudi apps Ø PVSS monitor panel for EFF. Ø Control. Mgr connects to Dim. H 1 D histos. Ø Subfarm functionality: • Add histograms from node together in one subfarm histogram. Ø Farmnode functionality: • PVSS control manager “pulls” each published value (float, int, string, histogram) at fixed time intervals. • Panel displays monitored values • Panel displays PVSS trend of monitored value ICALEPCS / Oct-03 Philippe Vannerem 16
Communicating histograms on-line GAUDI app SCADA app ROOT app Ø Generalized concept of updated online histograms with DIM. Ø Uses the AIDA interface classes for histograms (GAUDI-independent). Ø Tested with ROOT histogram viewer. Ø ROOT subscribes as a client to a Dim. H 1 D histogram, served by a Gaudi application, PVSS, … ICALEPCS / Oct-03 Philippe Vannerem 17
Conclusions Ø The LHCb design philosophy of a fully integrated hierarchical Experiment Control System is applied both to control hardware and software devices. Ø GAUDI trigger algorithms running on the event filter farm will be controlled and monitored in a distributed way with a SCADA system. Ø The interfacing package GAUCHO facilitates steering and monitoring of trigger algorithms for supervision and data quality checking purposes in an on-line environment. philippe. vannerem@cern. ch ICALEPCS / Oct-03 Philippe Vannerem 18
- Slides: 18