New WLCG Transfers Dashboards WLCG FTS XRoot D
New WLCG Transfers* Dashboards * (WLCG, FTS, XRoot. D) Magnoni Luca (IT/CM/MM) 5/17/17 Monit Update @ ADC Analytics 2
MONIT Architecture: quick recap 5/17/17 Monit Update @ ADC Analytics 3
MONIT Transfers Dashboards • Goal: provide user-experience and features comparable with existing dashboards, or better • Main Technologies for this work: • • • Kafka/Spark jobs to transform and enrich Grafana for visualization Influx. DB as storage backend 5/17/17 Monit Update @ ADC Analytics 4
Old Transfers Dashboards • WLCG: overview of data transfers and accesses for all experiments • • • FTS: • • • http: //dashb-wlcg-transfers. cern. ch derived from FTS and XRoot. D statistics overview of FTS transfers http: //dashb-fts-transfers. cern. ch XRoot. D: • • overview of XRoot. D data accesses http: //dashb-wdt-xrootd. cern. ch WLCG used as example here, same applies to FTS and XRoot. D 5/17/17 Monit Update @ ADC Analytics 5
On the new dashboards 5/17/17 Monit Update @ ADC Analytics 6
5/17/17 7
Main Features • • • Dropdown selection menus Dynamic Group-By Custom Filters Time Zooming on plot Custom Binning Computed Metrics: • • • Throughput (data transferred / time bin) Efficiency (for FTS) (success / failures)% Embeddable plots (iframe) 5/17/17 Monit Update @ ADC Analytics 8
User Experience • Get a responsive view on live data responsive ~ few seconds to load, live ~ few minutes ago • • Browse historical data weeks/months/years few tens of seconds to load drill down on interesting area with in-plot zooming • • Speed-Up Tip: choose a bin size appropriate to the length of the interval selected (twice bin size -> half loading time…) • • e. g. 1 d is a good choice for a several months plot 5/17/17 Monit Update @ ADC Analytics 9
Resolution and Data Retention • Transfers statistics have different resolutions and different retentions: • • 10 minutes resolution kept for 1 month 1 hour resolution kept for 5 years Leveraging on Influx. DB automatic aggregation capabilities • Historical data for 2016, 2017 has been imported from old dashboards • 5/17/17 Monit Update @ ADC Analytics 10
The matrix Not provided by any plugins : ( • Next Grafana (4. 3) comes with histograms and better heatmap • • • useful, but not really a matrix We’re considering Grafana #7169 contribution • to develop a matrix-like feature in the official table plugin and push upstream (more news soon…) 5/17/17 Monit Update @ ADC Analytics 11
5/17/17 12
On Alarms (news!) Together with the usual MONIT processing platform (Kafka/Spark, etc. ) • MONIT now also support alarms from Grafana 4 • If you can plot it, you can alarm it ! • 5/17/17 Monit Update @ ADC Analytics 13
Alarms on Grafana • Users can create a threshold-rule on a plot via the Grafana UI • • Get a notification when the rule triggers • • mail, mattermost, soon GNI/SNOW Good for simple use cases and basic aggregation • • rule periodically evaluated by Grafana server e. g. get a mail when throughput from A to B below limit… Works with data on Influx. DB 5/17/17 Monit Update @ ADC Analytics 14
On Alarms (news!) 5/17/17 15
Demo Time (WLCG, FTS, XRoot. D)
Summary • New WLCG Transfers dashboards • WLCG, FTS, XRoot. D • Based on Kakfa/Spark, Grafana and Influx. DB MONIT integrated, hence data in Elasticsearch/Kibana and HDFS too for exploration and analytics • 5/17/17 Monit Update @ ADC Analytics 17
MONIT possibilities: LHCOPN/WLCG traffic comparison 5/17/17 Monit Update @ ADC Analytics 18
Next Steps • Update public displays (e. g. Google Earth, WLCG home) • Integrate your feedback (monit-support) • Redirect old dashboard URLs to new • Keep old workflow accessible till end 2017 • Retire old dashboards early 2018 5/17/17 Monit Update @ ADC Analytics 19
Thank you! Docs: https: //cern. ch/monitdocs • Support: https: //cern. ch/monit-support • 5/17/17 Monit Update @ ADC Analytics 20
- Slides: 21