BNL Service Challenge 3 Site Report Xin Zhao

BNL Service Challenge 3 Site Report Xin Zhao, Zhenping Liu, Wensheng Deng, Razvan Popescu, Dantong Yu and Bruce Gibbard USATLAS Computing Facility Brookhaven National Lab

Services at BNL T FTS (version 2. 3. 1) client + server and its backend Oracle and myproxy servers. q FTS does the job of reliable file transfer from CERN to BNL. q Most Functionalities were implemented. It became reliable in controlling data transfer after several rounds of redeployments for bug fixing: short timeout value causing excessive failures, incompatibility with d. Cache/SRM. q Does not support DIRECT data transfer between CERN to BNL d. Cache data pool server (d. Cache SRM third party data transfer). The data transfers actually go through a few d. Cache Grid. FTP door nodes at BNL, which presents scalability issue. We had to move these door nodes to non-blocking networking ports to distribute traffic. q Both BNL and RAL discovered that the number of streams per file could not be more than 10, (a bug)? T Networking to CERN: q Network for d. Cache was upgraded to 2*1 Gpbs around June. q Shared link with Long Round Trip Time: >140 ms, while RTT for Europe sites to CERN is about 20 ms. q Occasional packet losses were discovered along the path between BNL-CERN. q 1. 5 G bps aggregated bandwidth observed by iperf with 160 TCP streams. 2

Services at BNL T d. Cache/SRM (V 1. 6. 5 -2, with SRM 1. 1 interface, Total 332 (3. 06 Ghz, 2 GByte Memory and 3 SCSI 140 Gbyte drives) nodes with about 170 TB disks, Multiple Grid. FTP, SRM, and d. Cap doors ): USATLAS production d. Cache system. q All nodes have Scientific Linux 3 with XFS module compiled. q Experienced High load on write pool serves during large amount data transfer. Was fixed by replacing the EXT file systems with XFS file system. q Core server crashed once. Reason was identified and fixed. q Small buffer space (1. 0 TB) for data written into d. Cache system. q d. Cache can now deliver up to 200 MB/second for input/output (limited by network speed. ) T LFC (1. 3. 4) client and server was installed at BNL Replica Catalog Server. q Server was installed. Tested the basic functionalities: lfc-ls, lfc-mkdir etc. q Will populate LFC with the entries in our production globus RLS server. T ATLAS VO Box (DDM + LCG VO box) was deployed at BNL. 3

BNL d. Cache Configuration DCap Clients Grid. FTP Clientsd Data Channel Oak Ridge Batch system HPSS SRM Clients Control Channel write pools Read pools DCap doors Grid. FTP doors Pnfs Manager Pool Manager SRM doors DCache System 4

CERN Storage System 5

Data Transfer from CERN to BNL (ATLAS Tier 1) 6

Transfer Plots Castor 2 LSF plugin problem 7

BNL SC 3 data transfer All data actually are routed through Grid. Ftp doors SC 3 Monitored at BNL 8

Data Transfer Status TBNL stablized FTS data transfer with high successful completion rate, as shown in the left image. TWe have attained 150 MB/second rate for about one hour with large number (> 50) of parallel file transfers. CERN FTS had the limit of 50 files per channel, which is not enough to fill up CERN BNL data channel. 9

Final Data Transfer Reports 10

Lessons Learned From SC 2 T Four file transfer servers with 1 Gigabit WAN network connection to CERN. T Meet the performance/throughput challenges (70~80 MB/second disk to disk). q Enabled data transfer between d. Cache/SRM and CERN SRM at openlab z Design our own script to control SRM data transfer. q Enabled data transfer between BNL Grid. Ftp servers and CERN openlab Grid. Ftp servers controlled by Radiant software. T Many components need to be tuned q 250 ms RRT, high packet dropping rate, has to use multiple TCP streams and multiple file transfers to fill up network pipe. q Sluggish parallel file I/O with EXT 2/EXT 3, lot of processes with “D” state, more file streams, worse the performance on file system. q Slight improvement with XFS system. Still need to tune file system parameter 11

Some Issues T Service Challenge also challenges resource: q Tuned network pipes, optimized the configuration and performance of BNL production d. Cache system and its associate OS, file systems, q Required more than one staff’s involvements to stabilize the newly deployed FTS, d. Cache and network infrastructure. q Staffing level decreased as services became stable. T Limited Resources are shared by experiments and users. q At CERN, SC 3 infrastructure are shared by multiple Tier 1 sites. z q Due to the heterogeneous nature of Tier 1 sites, data transfer for each site should be optimized non-uniformly based on site’s various aspects: i. e. network RRT, packet loss rates, experiment requirements etc. At BNL, network and d. Cache are also used by production users. z Need to closely monitor the SRM and network to avoid impacting production activities. T At CERN, James Casey alone handles answering email, setting up the system, reporting problems and running data transfer. He provides 7/16 support himself. q How to scale to 7/24 production support/production center? q How to handle the time difference between US and CERN? q CERN Support Phone (Tried once, but the operator did not speak English) 12

What have been done. T SC 3 Tier 2 Data Transfer q Data were transferred to three selected Tier 2 sites. T SC 3 Tape Transfer q Tape Data Transfer was stablized at 60 MB/second with loaned tape resources. q Met the goal defined at the beginning of Service Challenge. q Full Chain of data transfer was exercised. 13

ATLAS SC 3 Service Phase 14

ATLAS SC 3 Service Phase goals T Exercise ATLAS data flow T Integration of data flow with the ATLAS Production System T Tier-0 exercise T More information: q https: //uimon. cern. ch/twiki/bin/view/Atlas/DDMSc 3 15

ATLAS-SC 3 Tier 0 T Quasi-RAW data generated at CERN and reconstruction jobs run at CERN q No data transferred from the pit to the computer centre T “Raw data” and the reconstructed ESD and AOD data are replicated to Tier 1 sites using agents on the VO Boxes at each site. T Exercising use of CERN infrastructure … q Castor 2, LSF T and the LCG Grid middleware … q FTS, LFC, VO Boxes T Distributed Data Management (DDM) software 16

ATLAS Tier-0 RAW ESD (2 x) AODm 1. 6 GB/file 0. 2 Hz 17 K f/day 320 MB/s 27 TB/day AODm (10 x) 0. 44 Hz 37 K f/day 440 MB/s EF 1 Hz 85 K f/day 720 MB/s T 1 T 1 castor 2. 24 Hz 170 K f/day (temp) 20 K f/day (perm) 140 MB/s 0. 4 Hz 190 K f/day 340 MB/s RAW AOD ESD AODm 0. 5 GB/file 0. 2 Hz 17 K f/day 100 MB/s 8 TB/day 10 MB/file 2 Hz 170 K f/day 20 MB/s 1. 6 TB/day 500 MB/file 0. 04 Hz 3. 4 K f/day 20 MB/s 1. 6 TB/day CPU 17

ATLAS-SC 3 Tier-0 T Main goal is a 10% exercise q Reconstruct “ 10%” of the number of events ATLAS will get in 2007 using “ 10%” of the full resources that will be needed at that time T Tier-0 q ~300 k. SI 2 k q “EF” to CASTOR: 32 MB/s q Disk to tape: 44 MB/s (32 for raw and 12 for ESD+AOD) q Disk to WN: 34 MB/s q T 0 to each T 1: 72 MB/s q 3. 8 TB to “tape” per day T Tier-1 (in average): q ~8500 files per day q At a rate of ~72 MB/s 18

ATLAS DDM Monitoring 24 h before 4 day intervention 29/10 - 1/11 We achieved quite good rate in the testing phase (sustained 20 -30 MB/s to three sites (PIC, BNL and CNAF). 19

Data Distribution T Use a generated “dataset” q Contains 6035 files (3 TB) and we tried to replicate it to BNL, CNAF and PIC. T BNL Data Transfer is under way. T PIC: 3600 files copied and registered q 2195 ‘failed replication’ after 5 retries by us x 3 FTS retries z Problem under investigation q 205 ‘assigned’ - still waiting to be copied q 31 ‘validation failed’ since SE is down q 4 ‘no replicas found’ LFC connection error T CNAF: 5932 files copied and registered q 89 ‘failed replication’ q 14 ‘no replicas found’ 20

General view of SC 3 T When everything is running smoothly ATLAS get good results T The middleware (FTS) is stable but there were still lots of compatibility issues: q FTS does not work new version of d. Cache/SRM (version 1. 3). q ATLAS DDM software dependencies can also cause problems when sites upgrade middleware T not managed to exhaust anything production s/w; LCG m/w) T Still far from concluding the exercise and not running stably in any way. T Exercise will continue adding new sites. 21

SC 3 re-run T We will upgrade BNL d. Cache OS to RHEL 4 and d. Cache to 1. 6. 6 starting Dec/07/2005. T We will add few more d. Cache pool nodes if the software upgrades did not meet our expectation. T FTS should be upgraded if the necessary fix to prevent channel blocking is ready before new year. T LCG BDII needs to report status of d. Cache, FTS. (before Christmas). T We would like to schedule a test period at the Beginning of January for stability and scalability. T Everything should be ready by January 9. T Re-run will start at January 16. 22

BNL Service Challenge 4 Plan T Several steps needed to set-up hardware or service (ex: choose, procure, start install, end install, make operational), starting at January, ending before the beginning of March. q LAN, Tape system. q FTS, LFC, DDM, LCG VO boxes and other base line sevices will be maintained with agreed SLA and supported by USATLAS VO. q Dedicated LHC d. Cache/SRM write pool which provides up to 17 Tera bytes storage (24 hour worth data). (to be done synchronized with LAN, WAN). T Deploy and strengthen necessary monitoring infrastructure based on ganglia, nagios, Monalisa and LCG-RGMA. (February). T Drill for service integration (March) q Simulate network failure, server crashes, and how support center will respond to the issues. q Tier 0/Tier 1 End-to-End high performance network operational: bandwidth, stability and performance. 23

BNL Service Challenge 4 Plan T April/2006, establish the stable data transfer in the speed of 200 M Bytes/second to disks and 200 M Bytes/second to tape. T May/2006, disk and computing farm upgrading. T July/01/2006: stable data transfer driven by ATLAS production system and ATLAS data management infrastructure between T 0~T 1 (200 M Bytes/second) and provide services to satisfy SLA (Service level agreement). T Details of involving Tier 2 are in planning too. (February and March) q Tier 2 d. Cache: UC d. Cache needs to be stabilize and operational in February, UTA and BU need to have d. Cache in March. q Base line client tools should be deployed at Tier 2 centers. q Base line services should support Tier 1~Tier 2 data transfer before SC 4 starts. 24