WLCG Transfers Dashboard A unified monitoring tool for

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche <Alexandre. beche@cern. ch> IT-SDC : Support for Distributed Computing

Outlines § § § Mission History of the data transfers monitoring tools Generic monitoring workflow New WLCG Transfers Dashboard Summary IT-SDC Alexandre Beche – White Area 18 December 2013 2

Mission ATLAS CMS FTS + XRoot. D LHCb ALICE FTS IT-SDC XRoot. D Alexandre Beche – White Area 18 December 2013 3

Mission ATLAS CMS FTS + XRoot. D LHCb ALICE FTS IT-SDC XRoot. D Alexandre Beche – White Area 18 December 2013 4

Mission ATLAS CMS FTS + XRoot. D LHCb ALICE FTS IT-SDC XRoot. D Alexandre Beche – White Area 18 December 2013 5

History of data transfers monitoring tools WLCG Transfers Dashboard WEB API / UI FTS WLCG IT-SDC Alexandre Beche – White Area 18 December 2013 6

History of data transfers monitoring tools WLCG Transfers Dashboard WEB API / UI XRoot. D integration IT-SDC Alexandre Beche – White Area AAA FAX FTS WLCG 18 December 2013 7

History of data transfers monitoring tools WEB API / UI WEB API/UI WLCG Transfers Dashboard XRoot. D integration XRoot. D Detailled Dashboard IT-SDC Alexandre Beche – White Area AAA FAX FTS WLCG 18 December 2013 8

History of data transfers monitoring tools WEB API / UI WEB API/UI WLCG Transfers Dashboard XRoot. D integration XRoot. D Detailled Dashboard IT-SDC Alexandre Beche – White Area EOS AAA EOS FAX FTS WLCG EOS integration 18 December 2013 9

History of data transfers monitoring tools WEB API / UI WEB API/UI WLCG Transfers Dashboard XRoot. D integration XRoot. D Detailled Dashboard WLCG FAX AAA EOS integration IT-SDC Alexandre Beche – White Area EOS AAA EOS FAX FTS ALICE integration 18 December 2013 10

Generic monitoring workflow AMQ IT-SDC Consumer Raw & Statistics Alexandre Beche – White Area WEB API 18 December 2013 WEB UI 11

From Active. MQ to Oracle § Load balanced collector § Horizontal scaling Consumer AMQ DB Consumer IT-SDC Alexandre Beche – White Area 18 December 2013 12

From Active. MQ to Oracle § Load balanced collector § Horizontal scaling Consumer AMQ DB Stompclt* * Developed by the CERN messaging team IT-SDC Alexandre Beche – White Area 18 December 2013 13

From Active. MQ to Oracle § Load balanced collector § Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* * Developed by the CERN messaging team IT-SDC Alexandre Beche – White Area 18 December 2013 14

From Active. MQ to Oracle § Load balanced collector § Horizontal scaling Consumer AMQ DB Stompclt* DB inserter Disk queue* § Customizable inserter § Data enhancement § Filtering * Developed by the CERN messaging team IT-SDC Alexandre Beche – White Area 18 December 2013 15

From Active. MQ to Oracle § Load balanced collector § Horizontal scaling Consumer AMQ DB Simplevisor* Stompclt* DB inserter Disk queue* § Modular architecture § Common building blocks (EPEL) § Reliable IT-SDC Alexandre Beche – White Area § Customizable inserter § Data enhancement § Filtering * Developed by the CERN messaging team 18 December 2013 16

Raw data aggregation § Well optimized storage § Partitioned table § No global indexes (including the PK) IT-SDC Alexandre Beche – White Area 18 December 2013 17

Raw data aggregation § Well optimized storage § Partitioned table § No global indexes (including the PK) § Aggregation of unordered events § Stateless: Full re-computation of touched bins each time § Compute stats from raw data in 10 min bins § Aggregate 10 min stats in daily bins IT-SDC Alexandre Beche – White Area 18 December 2013 18

Transfers Aggregation methods 2 pm IT-SDC 3 pm 4 pm 5 pm 6 pm Alexandre Beche – White Area 7 pm 18 December 2013 19

Transfers Aggregation methods Easy method 2 pm IT-SDC 3 pm 4 pm 5 pm 6 pm Alexandre Beche – White Area 7 pm 18 December 2013 20

Transfers Aggregation methods Easy method 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm Transfer s 1 0 0 2 1 Bytes 10 0 0 15 20 IT-SDC Alexandre Beche – White Area 18 December 2013 21

Transfers Aggregation methods Easy method 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm Transfer s 1 0 0 2 1 Bytes 10 0 0 15 20 Transfers 1 (1) 1 (0) 2 (0) 3 (2) Bytes 1 14 (9+6) 15 (1+9+5) 5 8 IT-SDC Alexandre Beche – White Area 1 (1) 18 December 2013 22

Transfers Aggregation methods Easy method 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm Transfer s 1 0 0 2 1 Bytes 10 0 0 15 20 Transfers 1 (1) 1 (0) 2 (0) 3 (2) Bytes 1 14 (9+6) 15 (1+9+5) 5 8 Adopted method 1 (1) • Both method equivalent if: • Small transfers or many transfers IT-SDC Alexandre Beche – White Area 18 December 2013 23

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Raw & Statistics DAO Collection of SQL queries Application specific Generic IT-SDC Alexandre Beche – White Area 18 December 2013 24

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Run a parameterized query, Return the results in a dict Raw & Statistics DAO Action Collection of SQL queries Application specific Generic IT-SDC Alexandre Beche – White Area 18 December 2013 25

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Run a parameterized query, Return the results in a dict Raw & Statistics Application specific DAO Collection of SQL queries Python dict to #format# converter Action Views Json, xml, … Generic IT-SDC Alexandre Beche – White Area 18 December 2013 26

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Raw & Statistics Application specific DAO Collection of SQL queries Python dict to #format# converter Action Views Json, xml, … Controller Run a parameterized query, Return the results in a dict Router of url+mime To action + view Generic IT-SDC Alexandre Beche – White Area 18 December 2013 27

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Raw & Statistics Application specific Generic IT-SDC DAO Collection of SQL queries Python dict to #format# converter Action Views Json, xml, … Configuration Alexandre Beche – White Area Controller Run a parameterized query, Return the results in a dict Router of url+mime To action + view List of actions 18 December 2013 28

A well defined web API § Legacy dashboard framework: § Use the MVC design pattern (based on Struts) Raw & Statistics Application specific Generic IT-SDC DAO Action Django under Collection of SQL queries Views investigation Controller Run a parameterized query, Return the results in a dict Router of url+mime To action + view Json, xml, … Python dict to #format# converter Configuration Alexandre Beche – White Area List of actions 18 December 2013 29

Visualization layer Dashboard UI § Rich web single-page user interface § AJAX+JSON communication IT-SDC Alexandre Beche – White Area 18 December 2013 30

Visualization layer Dashboard UI § Rich web single-page user interface § AJAX+JSON communication § j. Query + popular plugin technology cocktail IT-SDC Alexandre Beche – White Area 18 December 2013 31

Visualization layer Dashboard UI § § Rich web single-page user interface AJAX+JSON communication j. Query + popular plugin technology cocktail Modular code reuse with xbrowse framework § Lightweight in-house URL-driven MVC (<300 LOC) § Views shared with several other applications § Common look and feel xbrowse IT-SDC Alexandre Beche – White Area 18 December 2013 32

History of data transfers monitoring tools WEB API / UI WEB API/UI WLCG Transfers Dashboard XRoot. D integration XRoot. D Detailled Dashboard WLCG FAX AAA EOS integration IT-SDC Alexandre Beche – White Area EOS AAA EOS FAX FTS ALICE integration 18 December 2013 33

WLCG Transfers Dashboard federated approach IT-SDC FAX AAA Alexandre Beche – White Area EOS ALICE AAA WEB API/UI EOS WEB API/UI FAX FTS WEB API ALICE WEB API / UI No duplication anymore EOS could be included 18 December 2013 34

WLCG Transfers Dashboard federated approach WLCG Transfers Dashboard API / UI IT-SDC FAX AAA Alexandre Beche – White Area EOS ALICE AAA WEB API/UI EOS WEB API/UI FAX FTS WEB API ALICE WEB API / UI No duplication anymore EOS could be included 18 December 2013 35

Some plots IT-SDC Alexandre Beche 27 May 2013 36

New WLCG Transfers Dashboard § Multi-threaded HTTP Call § Thread pool § Async request (std: : future c++11) Aggregate Main thread Get results Return Async call 1 Async call 2 IT-SDC Alexandre Beche – White Area 18 December 2013 37

Not just a re-writing § Possible re-think of UI IT-SDC FAX EOS FTS WEB API/UI FAX WEB API / UI Alexandre Beche – White Area 18 December 2013 38

Not just a re-writing § VO specific UI IT-SDC FTS Alexandre Beche – White Area FAX FTS FAX EOS FTS WEB API/UI FAX WEB API / UI ATLAS Monitoring UI 18 December 2013 EOS § Possible re-think of UI 39

Summary § More flexible architecture § Database-less § Based on common building blocks § UI code fully re-used Thanks for your attention IT-SDC Alexandre Beche – White Area 18 December 2013 40

Useful links § AAA Dashboard § § FAX Dashboard: § § http: //dashb-fts-transfers/ui/ WLG Transfers Dashboard: § § http: //dashb-atlas-xrootd-transfers. cern. ch FTS Dashboard: § § http: //dashb-cms-xrootd-transfers. cern. ch http: //dashb-wlcg-transfers-new/ui/# Xbrowse framework: § https: //twiki. cern. ch/twiki/bin/view/Arda. Grid/Xbrowse. Framework IT-SDC Alexandre Beche - ITTF 29 November 2013 41