Enabling Grids for Escienc E VOspecific systems for

Enabling Grids for E-scienc. E VO-specific systems for the monitoring of the LHC computing activities on the GRID Julia Andreeva, CERN (IT/GS) NEC 09, September 2009, Varna, Bulgaria www. eu-egee. org EGEE-III INFSO-RI-222667 EGEE and g. Lite are registered trademarks

Outline Enabling Grids for E-scienc. E Ø Monitoring from the VO perspective, motivation Ø Overview of the existing VO-specific monitoring systems and their role in operating of the WLCG infrastructure Ø Experiment Dashboard as an example of the common application used by all LHC VOs. Ø High-level cross-VO view based on data from VOspecific systems Ø Summary EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 2

Monitoring from the VO perspective- why is it so important? Enabling Grids for E-scienc. E Ø We are still accumulating practical experience operating Grid infrastructure Ø We are not yet aware of all possible troubles which could happen to the infrastructure as a whole or to individual components Ø As a consequence, we do not yet have enough knowledge to create perfect monitoring system which would alarm us in any critical situation or even better would predict such situation before it happens Ø All this implies considerable involvement of the user community to the operations Ø As current experience shows (CCRC 08, STEP 09 and beyond) as a rule VO communities, in particular people taking computing shifts, are those who detect problems in the first place Ø VO monitoring tools are the main monitoring instrument for the moment. They are aggregating and promptly adapting new experience in operating of the Grid infrastructure EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 3

Main areas covered by the VO monitoring tools Enabling Grids for E-scienc. E Ø Job processing (sharing and usage of the resources, performance, reasons of the failures and correspondingly related problems with the involved Grid services or VO applications) Ø Data transfer (throughput, efficiency, reasons for the failures and related problems with the involved Grid services) Ø Overall status of sites serving a given VO (site commissioning , computing shifts) EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 4

Variety of tools used by the LHC VOs Enabling Grids for E-scienc. E Ø Ø - ALICE Mon. Alisa for job processing Mon. Alisa and Experiment Dashboard for data transfer ATLAS Panda and Experiment Dashboard for job processing Experiment Dashboard for data transfer CMS Prod. Mon and Experiment Dashboard for job processing Phedex for data transfer LHCb Dirac for job processing Dirac for data transfer All experiments are using - SAM and Experiment Dashboard for monitoring of site status and status of the services at the sites SLS for monitoring of services at Tier 0 EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 5

ALICE example Enabling Grids for E-scienc. E Monitoring system of ALICE based on Mon. Alisa monitoring systems. Monalisa services at all ALICE sites for site-level monitoring + Mon. Alisa repository for a high-level view on the scope of the ALICE VO EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 6

Monitoring of ATLAS DDM Enabling Grids for E-scienc. E Monitoring of ATLAS DDM is implemented in Dashboard framework. The information sources are ATLAS DDM services at the sites. Data repository is implemented in ORACLE backend located at CERN. Widely used by ATLAS community. Up to 1 K unique visitors per month, More that 100 K pages are viewed daily. INFSO-RI-222667 EGEE-III NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 7

CMS example Enabling Grids for E-scienc. E Monitoring of the CMS transfers is coupled with the CMS Data distribution system Ph. EDEX. Provides information about transfer rate, transfer quality, the status of the queue for transfer requests, etc… For CMS Job Monitoring see next talk of Irina Sidorova EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 8

Monitoring of the LHCb computing activities by Dirac Enabling Grids for E-scienc. E In LHCb both Data transfer and job monitoring are provided by Dirac EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 9

Experiment Dashboard as an example of system used by 4 LHC VOs Enabling Grids for E-scienc. E ØExperiment Dashboard is in production for 4 LHC VOs ØWidely used by the experiments for their everyday work (3 K unique visitors (unique IP addresses) of the CMS production server in August 2009) ØCovers full range of the LHC computing activities ØWorks transparently across various Grid infrastructures ØDeveloped as a result of the joined effort of the Dashboard team , developers in the LHC experiments and in other monitoring projects. In collaboration with institutes from Taiwan, Russia, France and Great Britain EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 10

Collaboration with JINR and other Russian institutions Enabling Grids for E-scienc. E Ø Russia is actively participating in the WLCG monitoring activity, namely contributing to the Dashboard project. From Russian side this work is coordinated by Vladimir Korenkov Ø Strong contribution from JINR: Irina Sidorova Elena Tikhonenko Sergey Belov Sergey Mitsyn Alexander Uzhinskiy Andrey Nechaevskiy Among our JINR colleagues there are many young developers recently graduated from Dubna University Ø Very much hope that this collaboration will continue EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 11

Enabling Grids for E-scienc. E Generic applications: • Job Monitoring • Task monitoring for the analysis users • Site availability based on SAM tests • Site Status Board EGEE-III INFSO-RI-222667 Experiment Dashboard applications VO-specific applications: • ALICE Data Transfer Monitoring • ATLAS Data Management Monitoring • ATLAS Production Monitoring • Central Repository for Production Monitoring Data for CMS NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 12

Development principles Enabling Grids for E-scienc. E Ø Do not develop and deploy new sensors unless nothing is in place for a given purpose Ø Where possible use common solutions (technology and implementation). All Dashboard applications regardless of their functionality and information sources are developed in the Dashboard framework Ø Involvement users in the development process EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 13

Enabling Grids for E-scienc. E Experiment Dashboard Framework s ) es O cc A A (D B r D aye L Data storage and aggregation g tin c a d ar o hb s s t Da gen A t Da le ol Machinereadable format publisher Other applications C DB Access Layer (DAO) Information sources Dashboard agents EGEE-III INFSO-RI-222667 DB Access Layer (DAO) UI System is modular This allows to have flexible approach while implementing needs of the customers NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 14

Enabling Grids for E-scienc. E Experiment Dashboard Framework (Examples) s ) es O cc A A (D B r D aye L Data storage and aggregation g tin c a d ar o hb s s t Da gen A t Da le ol Machinereadable format publisher Other applications C DB Access Layer (DAO) Information sources Dashboard agents EGEE-III INFSO-RI-222667 DB Access Layer (DAO) UI For ATLAS Data Management Monitoring all components are implemented NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 15

Enabling Grids for E-scienc. E Experiment Dashboard Framework (Examples) s ) es O cc A A (D B r D aye L Data storage and aggregation g tin c a d ar o hb s s t Da gen A t Da le ol Machinereadable format publisher Other applications C DB Access Layer (DAO) Information sources Dashboard agents EGEE-III INFSO-RI-222667 DB Access Layer (DAO) UI For CMS Production Monitoring Dashboard is used to store, aggregate and archive data and to publish it in XML format. While UI is developed by the CMS Production Team NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 16

For new SAM portal information is not imported into Dashboard DB. In SAM DB some additional tables are created and availability calculations are implemented inside ORACLE SAM instance. UI s ) es O cc A A (D B r D aye L Data storage and aggregation g tin c a d ar o hb s s t Da gen A t Da le ol DB Access Layer (DAO) Enabling Grids for E-scienc. E Experiment Dashboard Framework (Examples) Machinereadable format publisher Other applications C DB Access Layer (DAO) Dashboard is used only to create monitoring display and to publish data in the machinereadable format Information sources Dashboard agents EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 17

Make users to take part in the development Enabling Grids for E-scienc. E Monitoring applications are successful when they are developed in the close collaboration with user community. Good examples are Site Status Board and Dashboard Site Availability application based on SAM tests ØCMS Experiment over last year put a lot of effort in site commissioning activity Dashboard plots demonstrating improvement of the quality of the sites used by CMS. ØMonitoring is a vital component of this process ØNew applications had been developed in close collaboration between Dashboard team and members of the CMS community involved in the site commissioning activity ØInitially developed for CMS, Dashboard Site Availability application had been requested by other LHC VOs. Now in production for all 4 LHC VOs. ØSame for Site Status Board , had been developed for CMS was later requested by ALICE and LHCb. EGEE-III INFSO-RI-222667

Make users to take part in the development Enabling Grids for E-scienc. E Site Status Board EGEE-III INFSO-RI-222667 • These are users (people taking part in computing shifts and site commissioning activity) who define the set of columns, their content, which metrics are considered for overall status of the site, what is the validity interval for a given metric, which columns should be shown in the UI by default , alternative views, etc… • Dashboard provides a framework to be filled in by the customized information. • Historical information as well as straight forward navigation to the primary information source is available

High level cross-VO view Enabling Grids for E-scienc. E Ø The VO-specific monitoring systems are working in the scope of a single experiment Ø Non-expert users or users external to a given VO do not know how to find required information Ø It is difficult if at all possible to compare and correlate information of different VOs. Global cross-VO view is missing Ø Recent development aims to solve this problem Ø The systems providing high level view are being designed. They are based on integration of the experiment-specific monitoring systems, Dashboard framework and Grid. Map visualization system EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 20

Grid. Map visualization system Enabling Grids for E-scienc. E ØGrid. Map visualization tool had been developed in the context of CERN Openlab collaboration between CERN and EDS company ØThe main motivation for Grid. Map development is to provide a high level view of the monitoring data collected from the distributed infrastructure in a intuitive and useful way. ØPerfect match of the requirements for visualization of the distributed hierarchical infrastructure and Grid. Map visualization EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 21

Use-cases for Grid. Map Enabling Grids for E-scienc. E ØMultiple use cases had been defined for Grid. Map : • Grid. Map for Experiment Work Flows • Grid. Map for status of services defined as critical by the LHC VOs • Grid. Map for Site Status Board ØAnalyzing the results of the CCRC 08 and STEP 09 one of the main conclusions was that sites are a bit disoriented regarding monitoring. Too many monitoring tools… Which ones to use? Which ones to trust? How to understand whether VOs served by the site are happy about site performance ? • Siteview application aims to provide estimation of site performance from the VO perspective • http: //dashb-siteview. cern. ch/gridmap-vo-siteview/ EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 22

High-level monitoring system for sites serving LHC VOs Enabling Grids for E-scienc. E ATLAS Grid Map for a particular site ALICE CMS • Central repository for common metrics • (transfer rate, parallel jobs, success rate, etc…) LHCb • Common metrics distributions by time EGEE-III INFSO-RI-222667 EGEE'08 - Julia Andreeva (CERN, IT/GS)

Siteview(1/4) Enabling Grids for E-scienc. E Map is split in 4 groups: • Overall status of the site from the VO perspective • Job processing activity • Incoming data transfer • Outgoing data transfer • Size of the cell is defined by the scale of a given activity, colour is defined by the success rate EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 24

Siteview (2/4) Enabling Grids for E-scienc. E Assists users to navigate to the primary information source EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 25

Siteview (3/4) Enabling Grids for E-scienc. E • Click to get more information about failures EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 26

Siteview (4/4) Enabling Grids for E-scienc. E • Click to get to the primary information source EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 27

Integration with Google Erath Enabling Grids for E-scienc. E Experiment specific monitoring systems provide input data Dashboard agents publish this information in the KML format Strong contribution to the development of Sergey Mitsyn (JINR) Application will be shown during the LHC demo at the EGEE conference in Barcelona EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 28

Summary Enabling Grids for E-scienc. E • Practical experience in operating Grid infrastructure and in using it by the LHC community (in particular during CCRC 08, STEP 09) proved that VO-specific monitoring systems are the vital part of the operations and are currently the main source of the monitoring information • Having a wide range of VO-specific monitoring tools in place, we were still missing the high level view of the computing activities for LHC experiments altogether both at the global and at the site level • This issues are being addressed in the current development. • Siteview application is being developed and evaluated by the LHC community EGEE-III INFSO-RI-222667 NEC 09, Varna, Julia Andreeva (CERN, IT/GS) 29