Job Monitoring for the LHC experiments Irina Sidorova

Job Monitoring for the LHC experiments Irina Sidorova (CERN, JINR) Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it on behalf of the Dashboard team

Table of content • Importance of Job Monitoring • Overview of the Dashboard Job Monitoring applications • Monitoring of user analysis • Conclusions Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Importance of Job Monitoring • Data distribution and data processing are two main computing activities for VOs running on WLCG infrastructure • Quality of job processing provides the estimation of the quality of the infrastructure in general and defines the overall success of the computing activities of the VOs • On the other hand, detailed and reliable job monitoring helps to improve the computing models of the LHC VOs Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Dashboard main goals The goal of the Experiment Dashboard is to monitor the activities of the LHC experiments on the distributed infrastructure, providing monitoring data from the virtual organization (VO)/user perspectives. The LHC experiments use various Grid infrastructures (LCG/EGEE, OSG, NDGF) with correspondingly various middleware flavors and job submission methods. Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it The main task is to provide a uniform and complete view of various activities like job processing, data movement and publishing, access to distributed databases regardless of the underlying Grid flavor.

Overview of the Dashboard job monitoring applications • Atlas Prod. Sys Monitoring • Central repository for CMS Prod. Agent monitoring data • Generic Job Monitoring • Monitoring of user analysis jobs Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Data flow for Dashboard Job Monitoring LB, CEMon, Condor-g, jobs instrumented to report their progress, Job Submission Tools of the experiments Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it Mon. ALISA, currently we’re going to switch to the Messaging System for the Grid (MSG) Data is available in various formats and can be presented for different categories of users: VO managers, computing shifters, MC production teams, Site commissioning, LHC physicists running their jobs on the Grid

• Dashboard job monitoring is designed as common solution for any virtual organization • To provide complete view of job processing both from the Grid and application point of view the VO job submission tools should be instrumented to report job’s status information • Dashboard job monitoring for CMS is the most advanced one since all CMS submission tools are well instrumented for the Dashboard reporting Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Dashboard Job Monitoring functionality • Interactive view Shows what is going on NOW. -Distribution of jobs by site, CE, user, submission tool, application version, dataset, etc… -Distribution of jobs by status -Success rate, CPU and wall clock time, number of processed events • Historical Interface Job statistics distributed over time • Dashboard Task Monitoring Provides complete information about analysis job processing. Serves the needs of the analysis community and of the analysis support team Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it • Quick Analysis of Error Sources Automatically detects failing grid components and offers solutions to solve the problems.

Dashboard Task Monitoring • Distributed user analysis on the WLCG infrastructure is currently the main challenge of the LHC computing • With data taking approaching number of analysis users will dramatically increase • User-friendly, complete and reliable monitoring of the analysis task processing is an important factor for successful organization of the distributed analysis • Task Monitoring application is developed on top of the common job monitoring repository • Main users of the application are LHC physicists, distributed analysis support teams and site administrators Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

CMS Task monitoring for analysis users Progress of processing in terms of processed events Distribution of jobs by their current status Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it Very detailed per job information Failure diagnostics for GRID and application failures Distribution of efficiency By site Provides transparent monitoring regardless submission method or middleware platform Detailed view of user tasks including failure diagnostics, processing efficiency and resubmission history Low latency, updates from the worker node where job is running User driven development 10

CMS Dashboard Usage by application • Application is currently in production for CMS VO • Became very popular in the CMS physics community • Got a very positive feedback from the users • Up to 150 physicists are using the application on everyday basis Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Trying to understand the failure reasons Ø VO jobs are instrumented to report the application exit codes. Unfortunately the exit codes are not always pointing to a particular failure reason. In most cases they are rather obscure Ø Failure of the job can be caused by many different reasons: • Error in the user code • Misconfiguration of the site -misconfiguration of the worker nodes on the site -corrupted distribution of the experiment software -problem of the accessing of the shared area from the worker node -etc. . . • Problem accessing input data • Problem saving output files to the remote storage Internet Services n CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it Ø In order to adress the problem it’s necessary to understand the underlying reason. Possible ways to achive this goal are: - better diagnostics in case of failure published from the user jobs - analysis of the failure statistics Ø collaboration with the developers of the workload management systems of the experiments Ø what went wrong with his job but can get all the necessary information from user interface.

Quick Analysis of Error Sources (QAOES) Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it • System of problem detection with the Association Rule Mining algorithm • Expert system • Aim is to decrease a time of fault detection and to improve grid reliability • QAOES prototype is in production for the CMS analysis job monitoring data • The tool is being evaluated by CMS distributed analysis support team

QAOES use case 1 Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

QAOES use case 1 Jobs overview on T 2_FR_IPHC site sorted by activity for the last 6 hours Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it User analysis jobs have very low success rate. Let’s see whether these jobs relate to one particular user or not.

Jobs overview on T 2_FR_IPHC site sorted by user for the last 6 hours Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it Almost all users have jobs failed from the application point of view. Let’s check why two users don’t have such problem.

QAOES use case 1 Jobs overview on T 2_FR_IPHC site sorted by dataset for the last 6 hours Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it Jobs with “unknown” dataset don’t use any input data. And the other jobs failed with 8020 error code. It’s a data access problem. We see that automatically generated rule correctly detected the faulty component – site. Namely, data access problem at the site.

QAOES use case 2 Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

QAOES use case 2 Distribution of the user jobs per site for the last 6 hours Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it A large number of jobs belonging to the user failed, cancelled or aborted on different sites. Let’s check if it happened with the particular task of the user or not. Sort the jobs by task.

Distribution of the user jobs per task Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it The user jobs fail on different sites with different tasks. It could be an input data problem. Let’s check.

Distribution of the user jobs per dataset Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it User is failing at various sites running different tasks and reading different datasets. It’s a clear indication of an error in the user code. Which is consistent with the automatically generated rule.

Add a solution Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it

Conclusions Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it • Monitoring of the job processing is one of the main indicators for estimation of the overall quality of the Grid infrastructure • User-friendly, reliable and complete monitoring is vital for effective organization of the distributed data analysis. • Developed in the close collaboration with the user community, Dashboard job monitoring applications provide required functionality for LHC offline computing activity • Future development and improvements are driven by the feedback and suggestions of the LHC users

Thank you for your attention! http: //dashboard. cern. ch Internet Services CERN IT Department CH-1211 Genève 23 Switzerland www. cern. ch/it