Constant monitoring of multisite network connectivity at the

Constant monitoring of multi-site network connectivity at the Tokyo Tier 2 center Tomoaki Nakamura ICEPP, The University of Tokyo 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 1

ATLAS completed jobs in Tier 2 Apr. 1, 2010 – Mar. 31, 2012 Analysis 1 2 3 4 5 6 7 8 9 10 2012/4/20 Production AGLT 2 MWT 2 SLAC TOKYO DESY-HH SWT 2_CPB QMUL LRZ FREIBURG DESY-ZN 8. 8 % 7. 7 % 5. 0 % 4. 9 % 4. 2 % 3. 5 % 3. 0 % 2. 8 % 2. 7 % Tomoaki Nakamura 1 2 3 4 5 6 7 8 9 10 AGLT 2 UKI-SCOTGRID-GLASGOW SLACXRD LRZ-LMU HU_ATLAS_Tier 2 DESY-HH UKI-NORTHGRID-MAN-HEP TOKYO UTA_SWT 2 UKI-LT 2 -QMul ICEPP, Univ. of Tokyo 6. 4 % 3. 9 % 3. 7 % 3. 2 % 2. 9 % 2. 6 % 2

FTS monitor (from Tier 1 s to TOKYO) Good 5 MB/sec Jan. 1 – Mar. 31, 2011 No good 5 MB/sec 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 3

FTS monitor (from TOKYO to Tier 1 s) Good 5 MB/sec Jan. 1 – Mar. 31, 2011 No good 5 MB/sec 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 4

TOKYO regional analysis center Network switch Tape robot PC node for local use Disk array Tier 2 Woker Nodes Disk array ~270 m 2 • Reduce power consumption by switching off many devices except for Tier 2 resources after the earthquake on Mar. 11, 2011 • Improve air-conditioning only for Tier 2 WNs 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 5

History of ATLAS regional analysis center in Tokyo Discussion in the ATLAS-Japan group on the possibility of having a regional computing center in Japan 1999: ICEPP, University of Tokyo decided to take the responsibility 2001~2005: R&D work 2005: Preparation of a computer room and an UPS room in School of Science Bldg. 1 2006: Installation of the first computer system (3 -year lease for 2007 -2009) 2009: Installation of the current system (3 year lease for 2010 -2012) 2012: Preparation of the new system. Procurement of the hardware in Summer Installation and upgrade by the end of 2012 In WLCG As a Tier 2 site (the only ATLAS site in Japan) Support ATLAS VO only For local use By ATLAS-Japan group (16 institutes, ~70 researchers, ~80 students) No Grid services (except UI function) 2012/4/20 Tomoaki Nakamura Managed by 5~6 FTEs System engineers (2 FTEs) from company (1~2 FTEs for Grid operations) Working on more software deployment GPFS, Castor ICEPP, Univ. of Tokyo 6

Current hardware configuration Blade servers (720 nodes: 5760 cores) DELL Power. Edge M 610 Dual 2. 8 GHz Nehalem (Xeon X 5560) CPUs (8 cores/node) 16 -24 Gbytes RAM 300 GB SAS HDDs (mirrored) 160 WNs are reserved for WLCG (1152 cores) 10 Gb. E NICs for 160 WNs Bandwidth 30 Gbps/16 nodes Disk arrays (120 units: 4. 8 PB) ~ 40 Tbytes/unit 24 x 2 Tbytes SATA HDDs 2 RAID-6 per one unit 8 Gbps Fibre-Channel connection 2 PB are reserved for WLCG File servers (64 nodes) DELL Power. Edge M 610 10 Gb. E NIC, Fibre Channel HBA (8 Gbps) 2 disk arrays/server Tape library unit Storage. Tek SL 8500 (Oracle) 8360 slots 22 LTO 3, 10 LTO 4, 5 LTO 5 drives accessed via 32 tape servers 2012/4/20 Network and others 2 large Ethernet core switches (Foundry RX-32) External WAN connection: 10 Gb. E to SINET 4 (NREN) One ORACLE RAC cluster (2 nodes) Miscellaneous machines (DNS servers, etc. ) Tomoaki Nakamura ICEPP, Univ. of Tokyo 7

Storage configuration for Tier 2 File Server: Dell Power. Edge M 610 CPU: Dual Intel Xeon E 5530 (Nehalem 2. 4 GHz, 4 cores/CPU). Memory: 24 Gi. B Fibre Channel HBA: Qlogic QME 2572 (8 Gbps) NIC： 10 Gbps (Broadcom 57710) Disk Array: Infortrend Eon. Stor S 24 F-G 1840 24 SATA 2 TB HDDs, RAID 6, 2 file system 8 Gbps Fibre Channel Interface Disk Array Fibre Channel Switch: Qlogic Sanbox 5800 8 G-FC 10 GE 10 Gbps Network Switch DPM: 68 XFS file systems (1. 36 PB) in one DPM pool (reserved up to 2 PB including spare) 2012/4/20 Tomoaki Nakamura File Server File System 8 G-FC Disk Array 8 G-FC x 17 ICEPP, Univ. of Tokyo File System 8

Internal network configuration RX-4 10 Gbps to WAN Big. Iron RX-32 x 2 8 x 10 Gbps perf. Sonar-PS: Latency 1 GE perf. Sonar-PS: Bandwidth 1 GE or 10 GE Non-blocking 10 Gbps All WNs and File servers for Tier 2 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 9

Internal bandwidth (WNs File servers) From file servers to WNs In Tokyo, jobs are saturated 4~6 K jobs are always queued 5, 000 MB/sec --To WNs: 30 Gbps x 9 enclosure Max: 33, 750 MB/sec ---3 K From WNs to file servers --- 2, 000 MB/sec 5 minutes averages To File servers 10 Gbps x 17 file servers Max: 21, 250 MB/sec Red: Production running Magenta: Analysis running Blue: Production queued Light Blue: Analysis queued 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 10

Routing between Tier 1 s and TOKYO http: //www. icepp. jp/monitor/bitraceroute/ Pacific TOKYO 10 Gbps TRIUMF Atlantic 10 Gbps NDGF US-LA ASGC BNL SARA NIKEF NII (NREN) is planning to extend SINET by one more 10 Gbps trans-Pacific line. Candidates: (depend on procurement) • One more 10 Gbps line to New York (man. Lan) (20 Gbps in total) • Another 10 Gbps line to Washington (WIX) 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo RAL CCIN 2 P 3 CERN CANF PIC 11

Connection in man. Lan CDF (Fermi) PHENIX (BNL) http: //dc-snmp. wcc. grnoc. iu. edu/manlan/ ATLAS (Tokyo Tier 2) ALICE (Hiroshima Tier 2) BELL II (KEK) ITER (France) Jan. 1, - Mar. 31, 2012 (1 day bin) GEANT 5 Gbps to GEANT to Man. Lan SINET 1 Gbps to SINET to Man. Lan http: //globalnoc. iu. edu/manlan/maps_documentation/diagrams. html 　 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 12

Data transfer throughput from/to Tier 1 s 2010 2011 ---800 MB/sec--- 2011 2012 [MB/sec] Incoming to TOKYO 1 day average Reached 800 MB/sec in 5 minutes average 2012/4/20 2011 2012 [MB/sec] Outgoing from TOKYO 2010 2011 Tomoaki Nakamura ICEPP, Univ. of Tokyo 13

Difficulties to find out the source of NW problems iperf May. 28, 2011 [Mbps] Sites security policy gridftp Projects, VOs computing model LYON TOKYO [MB/sec] Network providers support area LYON TOKYO LYON 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo May. 28, 2011 Large Medium Small 14

Case with CC-IN 2 P 3 Misconfiguration on the LBE packet related to Qo. S Nov. 17, 2011 packet loss cleared Nov. 17, 2011 LYON TOKYO LYON Improved by LAN reconfiguration in CC-IN 2 P 3 Jan. 21, 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 15

Comparing with the other sites LYON→TOKYO Large file [MB/sec] Medium file Small file BNL→TOKYO Large file [MB/sec] Medium file 2012/4/20 Small file Tomoaki Nakamura ICEPP, Univ. of Tokyo 16

Monitoring at TOKYO File by file transfer rate by grid. FTP log (TOKYO↔Tier 1 s) Data transfer test in French cloud (Tier 2↔Tier 2) Change of routing (TOKYO↔Tier 1 s) 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 17

Multi-site network monitoring Motoring at TOKYO for all ATLAS Tier 1 s and LHCONE sites http: //psps. perfsonar. net/ http: //lhcopn. web. cern. ch/lhcopn/ https: //twiki. cern. ch/twiki/bin/view/LHCOPN/Perfsonar. PS http: //lhcone. net/ https: //twiki. cern. ch/twiki/bin/view/LHCONE/Site. List 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 18

Full-meshed service monitoring (BNL) https: //130. 199. 185. 78: 8443/exda/? page=25&cloud. Name=LHCONE Throughput 2012/4/20 Tomoaki Nakamura Latency ICEPP, Univ. of Tokyo 19

Connectivity between TOKYO and Tier 1 s→TOKYO Very good Good enough 200 Mbps [Mbps] 600 Mbps TOKYO→Tier 1 s 600 Mbps Good enough 200 Mbps 2012/4/20 [Mbps] Very good Tomoaki Nakamura ICEPP, Univ. of Tokyo 20

perf. Sonar-PS: Good ASGC→TOKYO→ASGC TRIUMF→TOKYO→TRIUMF BNL→TOKYO→BNL 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 21

perf. Sonar-PS: Stable NDGF→TOKYO→NDGF CNAF→TOKYO→CNAF CCIN 2 P 3→TOKYO→CCIN 2 P 3 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 22

Perf. Sonar-PS: Need to improve RAL→TOKYO→RAL LRZ_DE→TOKYO→LRZ_DE PIC→TOKYO→PIC 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 23

Summary We will replace almost whole system by the end of this year. Expectation for the next system (at the present precision for Tier 2) WN: 1, 152 → 2, 304 core (16 core, 64 GB memory, 5 Gbps network per node) DPM Storage: 1. 36 PB (2 PB reserve) → 3 PB reserve File servers: 8 Gbps(FC), 10 Gbps(NIC) / 80 TB → 8 Gbps(FC), 10 Gbps(NIC) / 66 TB Capability of TOKYO will be increased much more. Therefore TOKYO should be integrated to T 2 D as soon as possible. I need Tier 1’s help to improve the network connectivity. Since stable bi-directional network connectivity among multi-sites is necessary for the smooth central operation of ATLAS computing, the continuous network monitoring at site level is one of the most important things. Perf. Sonar is very useful for the monitoring of the WAN performance and the comparison of multi-site connectivity to identify the source of network problems. Organization of the perf. Sonar deployment and preparation of fullmeshed service monitoring are very nice and well working. 2012/4/20 Tomoaki Nakamura ICEPP, Univ. of Tokyo 24