Monitoring Of XRoot D Federation A Petrosyan D

Monitoring Of XRoot. D Federation A. Petrosyan, D. Oleynik, J. Andreeva “Creating federated data stores for the LHC” workshop, 13. 09. 12, IN 2 P 3, Lyon

Historical overview • XRoot. D federation monitoring architecture was presented at Lyon last November, during previous “Creating federated data stores for the LHC” workshop – Motivation – Metrics – Requirements – Technologies • For details refer to https: //indico. in 2 p 3. fr/get. File. py/access? contrib. Id=14&res. Id= 1&material. Id=slides&conf. Id=5527 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 2

Components and technologies • Messages from XRoot. D servers are being collected via UCSD UDP collector and then being sent into AMQ • After being read from AMQ, data is stored in Hbase storage • Hadoop processing to prepare data summaries • Web-services for data export • Dashboard XBrowse transfer interface as UI 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 3

Why? • Why UCSD UDP collector? – Feds are ready to install this collector – Provides all necessary functionality – Supported • Why AMQ? – AMQ is a messaging system which is used for the monitoring purposes on the WLCG infrastructure – Is used in production by various applications including SAM – Message brokers supported by CERN IT • Why Hadoop? – Free – Scalable – Provides good data processing capabilities for log-like data • Why XBrowse? – – 13. 09. 2012 Interface created to show transfers Current UI of ATLAS DDM monitoring: http: //dashb-atlas-data. cern. ch/ddm 2/ Extendable Supported "Creating federated data stores for the LHC", IN 2 P 3, Lyon 4

Data flow for the XRoot. D federation monitoring 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 5

UDP messages collector • • Implemented by Matevz Tadel Can be installed anywhere, implemented as Linux daemon Sends complete transfer message to Active. MQ Message includes: – – – – 13. 09. 2012 Domain from, host and ip address Domain to, host and address User info File, size Bytes read/written Time transfer started/ended Num of operations Ect. "Creating federated data stores for the LHC", IN 2 P 3, Lyon 6

AMQ 2 Hadoop collector • Can be installed anywhere, implemented as Linux daemon • Listens Active. MQ queue • Extracts messages • Inserts into Hbase raw table 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 7

Hadoop processing • Reads raw table • Prepares data summary: 10 min stats as structure: – – – From To Sum bytes read Sum bytes written Amount files read Amount files written • Inserts summary data into summary table • Map. Reduce: we use Java, we also working on enabling Pig routines 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 8

Storage 2 UI data export • Web-service • Extracts data from the storage • Feeds Dashboard XBrowse UI 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 9

XBrowse UI • • • Implemented by David Tuckett Standalone web application Can be installed everywhere Accepts data in JSON format Presents transfers data as site matrix 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 10

13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 11

Status • Full prototype from UDP collecting to UI using simulated data is implemented in Dubna: http: //xrdfedmondev. jinr. ru/ui/#date. from=201206210000&date. interval=0&date. to =201206220000&grouping. dst=(host)&grouping. src=(host) • Basic functionality provides source/destination plots and matrix with a possibility to show local access/transfers versus remote, copying versus reading • Now migrating to CERN and real ATLAS and CMS data – Installation of Hadoop cluster at CERN (accomplished) – Setting up AMQ collector and Map. Reduce processing with real ATLAS and CMS data (in progress) 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 12

Nearest plans • Finish migration from JINR to CERN • Move from prototype to production in data processing • Start real data (ATLAS, CMS federations) consuming • Further development will follow the ATLAS and CMS requirements, see presentation of Rob Gardner: https: //twiki. cern. ch/twiki/pub/LCG/WLCGTransfer. Monitoring/Requirements. pdf 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 13

Thanks for attention 13. 09. 2012 "Creating federated data stores for the LHC", IN 2 P 3, Lyon 14