Controls ENICE Fw System Overview Tool FWWG Meeting

Controls EN-ICE Fw System Overview Tool FWWG Meeting 17 th Nov 09 P. Macuda, F. Varela

Controls Outline EN-ICE § Concept of the tool § Components and their functionality § What’s new § Demo 17. 11. 2009 2

Controls Concept EN-ICE Collaboration b/w LHCb and EN-ICE: • LHCb: Linux FMC agents • EN-ICE: the rest

Controls Architecture EN-ICE Windows PC: PVSS 00 procmon (FMC) Windows PC: Linux /Windows PC: System. Overview FMC Linux PC: FMC agents Linux PC: IPMI Server (FMC) Linux PC: FMC agents Legend: DIM WMI IPMI

Controls Components EN-ICE § System Overview Core • • Coherent views of all info gathered by the subcomponents Commands protected by Access Control Alarm handling Archiving § Pmon • Monitoring of managers § Farm Monitoring and Control • IPMI • Farm and Process monitoring § System Integrity (UNICOS) • Monitoring within the application, e. g. RDB connection, OPC serverclient connection

Controls Functions and constraints EN-ICE § System Overview is highly modular • User can decide what features (s)he wants to enable/add. Functionality Constraints Pmon None (pmon port, user, password) IPMI DIM server running on a central Linux machine Farm and process monitoring Linux – FMC DIM agents install on every PC Windows – PVSS 00 procmon running on a central PC and access rights on the remote PCs Access to info gathered by System Integrity Distribution connection to remote system(s) PVSS System Statistics Distribution connection to remote system(s)

Controls What’s new: System Overview Core EN-ICE Work done since an early version was presented to the experiments (only PMON functionality included) § Code Consolidation (deep review of the code performed) • New datapoint structure that allows for coherent navigation b/w the monitored items • Simplification of monitoring routines (easier maintenance) • Improved thread handling -> Stability improvements § Improvements • Enhancements to access control • More flexible alarm handling schema • Improved handling of embedded panels § New functionality • • • Integration of FMC and System Integrity information More flexible way to configure the tool: System Configuration DB or Interactive Commands on sets of managers in multiple projects, e. g. change settings Advanced manager search with filtering options Statistics on remote systems (# manager, #data points, # configs and lic. info)

Controls What’s new: FMC EN-ICE § Unification of DIM namespaces for Linux and Windows servers § Windows: PVSS 00 procmon • Central process that monitors remote computers over WMI ⁻ CPU, Memory, File System, Network, OS, Processes • Possibility to perform continuous monitoring of individual processes/services • It can be started as a PVSS manager (service), i. e. monitored by PMON § Linux: • • Implementation of summary and detailed services Changes to the RPMs and initial configuration Improvements on the IPMI server, e. g. password not visible in cmd line, etc. New version of process monitoring server ⁻ Possibility to perform continuous monitoring of individual processes

Controls Demo: LHCb setup EN-ICE §Monitoring of managers: • # Projects: 158, # Systems: 158, # Hosts: 140 §FMC • # Hosts: 799 (currently only IPMI configured for all nodes) • # IPMI sensors: 3921 Fans, 3104 temps, 539 V • OS monitoring (CPU, Mem, Procs, etc) currently only configured for a reduced set of nodes Work done by A. Sambade and A. Mazurov

Controls 17. 11. 2009 EN-ICE 10

Controls 17. 11. 2009 EN-ICE 11

Controls 17. 11. 2009 EN-ICE 12

Controls 17. 11. 2009 EN-ICE 13

Controls 17. 11. 2009 EN-ICE 14

Controls 17. 11. 2009 EN-ICE 15