File Stager test results Outline Single job performance

File. Stager test results Outline § Single job performance • Network performance Max Baak & Matthias Schott 1/29

Test setup Five file-transfer configurations: § Local disk (no file transfer) § File. Stager • Effectively: running over files from local disk § Xrootd • Buffered • Non-buffered § Rfio Files used: § Z ee, Z mumu AOD collections § ~37 mb/file, 190 kb/event § 200 events per File Max Baak & Matthias Schott 2/29

Intelligent File. Stager § cmt co –r File. Stager-00 -00 -19 Database/File. Stager • https: //twiki. cern. ch/twiki/bin/view/Main/File. Stager § Intelligent file stager copies files one-by-one to local disk, while running over previous file(s). • Run semi-interactive analysis over files nearby, eg. on Castor. • File pre-staging to improve wall-time performance. • Works in ROOT and in Athena. § Actual processing over local files in cache = fast! • • Only time loss due to staging first file. In many cases: prestaging as fast as running over local files! Minimum number of network connections kept open. Spreads the network load of accessing data over length of job. Max Baak & Matthias Schott 3/29

Large Scale Tests § Basic Test Setup • • Number of Files: 259 Total Datavolume: 9. 55 GB Number of Events: 52. 000 Only one job is execute on a batch machine Setup 2: Data Processed per [s] Series 1, XROOTD (NOT Buffered), 8. 92 E+05 Series 1, XROOTD (Buffered), 2. 00 E+06 Series 1, RFIO, 3. 26 E+05 Series 1, File Stager, 2. 71 E+06 § Three different Tests • Setup 1: Read only Event Number • Setup 2: Read 7 containers • Setup 3: Read 7 containers + Monte Carlo Truth Information + some Algorithmic • Setup 4: Same as Setup 2 but with Tag-File Access § Note • In this test the file access is overstressed as data-AOD files have a supposed file-size of 2 GB and contain much more events Max Baak & Matthias Schott Series 1, Local , 1. 64 E+06 Setup 3: Data Processed per [s] Series 1, XROOTD (NOT Buffered), 4. 53 E+05 Series 1, XROOTD (Buffered), 6. 12 E+05 Series 1, RFIO, 3. 81 E+05 Series 1, File Stager, 6. 58 E+05 Series 1, Local , 5. 72 E+05 4/29

Data processed per second § Data read per second, as measured by root. § All files cached on disk. Data format, program Reading speed [MB/s] AOD (7 container), Athena 2. 19 - 2. 32 AOD (7 containers), ARA 3. 75 AOD (trk. particles), Athena 2. 75 Vector<vector<>>, ROOT 4. 93 Simple ntuple, ROOT 6. 99 ALICE esd file 18 ROOT dummy example 47 Max Baak & Matthias Schott 5/29

Large Scale Tests Time in Seconds [s] § Reading Only Event Number • Reading 10% of File Content § Timing: • Comparable timing for local access, xrootd and file stager • RFIO 5 times slower Series 1, XROOTD (NOT Buffered), 699 Series 1, XROOTD (Buffered), 455 Series 1, RFIO, 2530 Series 1, File Stager, 519 Series 1, Local , 401 § Datatransfer • RFIO: 45 x larger data transfer than needed • xrootd (not buffered): 1% overhead of file transfer • xrootd (buffered): 32 x larger data transfer than needed • File. Stager: Copies whole file and hence 12 x larger data transfer than needed Max Baak & Matthias Schott Datatransfer in Byte Series 1, XROOTD (NOT Buffered), 7. 92 E+08 Series 1, XROOTD (Buffered), 2. 57 E+10 Series 1, RFIO, 3. 61 E+10 Series 1, File Stager, 1. 01 E+10 Series 1, Local , 7. 10 E+06 Series 1, Root Read Data Volume, 7. 88 E+08 Series 1, Total Data Volume, 9. 55 E+09 6/29

Read-ahead buffer § After read request, read ahead YYY kb • In anticipation of next read request. § Read-ahead buffer transferred and stored in cache. Castor: 1 Event 1 kb • Read-ahead: 512 kb • Cache size = 10 mb § Rfio: • Read-ahead: 128 kb • Cache size = read-ahead size (128 kb) Ø Effectively not used. § Unfortunately: caching not very successful for our purposes File Transfer § Xrootd: 1 kb Cache 128 kb Athena § Xrootd: Read-ahead buffer can be turned off. § Rfio: can probably be turned off as well, but we didn’t manage. Max Baak & Matthias Schott 7/29

Typical read access pattern bytes / read access § Typical AOD read access pattern. § Average: ~1 kb / read access § Note: 128 kb read-ahead buffer Max Baak & Matthias Schott 8/29

Large Scale Tests § Analysis I: • Reading 20% of File Content § Timing: • RFIO 5 times slower • Xrootd (not buffered) twice as slow as local access • File Stager faster than local access, as files are still in cache when loaded by Athena § Datatransfer • RFIO: 57 x larger data transfer than needed • xrootd (not buffered): 5% overhead of file transfer • xrootd (buffered): 22 x larger data transfer than needed • File. Stager: 5. 5 x larger data transfer than needed Max Baak & Matthias Schott Time in Seconds [s] Series 1, XROOTD (NOT Buffered), 1988 Series 1, XROOTD (Buffered), 885 Series 1, RFIO, 5515 Series 1, File Stager, 663 Series 1, Local , 1099 Series 1, XROOTD (NOT Buffered), 1, 892 E+06 Datatransfer in Byte Series 1, XROOTD (Buffered), 3, 993 E+07 Series 1, RFIO, 1, 029 E+08 Series 1, File Stager, 1, 015 E+07 Series 1, Local , 7, 754 E+03 Series 1, Root Read Data Volume, 1, 798 E+06 Series 1, Total Data Volume, 9, 551 E+06 9/29

Large Scale Tests Time in Seconds [s] § Analysis II: • Reading 35% of File Content and more algorithmic inside the analysis Series 1, XROOTD (NOT Buffered), 7170 Series 1, XROOTD (Buffered), 5300 Series 1, RFIO, 8590 § Timing: • Overall comparable timing as algorithmic part gets dominant • Xrootd (not buffered) is 20% faster than RFIO. • File Stager faster than local access, as files are still in cache when loaded by Athena § Datatransfer • Similar to previous analysis Max Baak & Matthias Schott Series 1, File Stager, 4970 Series 1, Local , 5720 Series 1, XROOTD (NOT Buffered), 3, 474 E+06 Datatransfer in Byte Series 1, XROOTD (Buffered), 4, 106 E+07 Series 1, RFIO, 1, 160 E+08 Series 1, File Stager, 1, 014 E+07 Series 1, Local , 0, 000 E+00 Series 1, Root Read Data Volume, 3, 272 E+06 Series 1, Total Data Volume, 9, 551 E+06 10/29

Large Scale Tests § Analysis I with Tag-Files: • Reading 20% of File Content • Access only 20% of the events in each file § Timing: • Local processing is the fastest (2 x faster than the next) • RFIO is dominated by latency of opening the files • Xrootd 3 x faster then rfio: § Datatransfer • Similar to previous analysis Time in Seconds [s] Series 1, XROOTD (NOT Buffered), 955 Series 1, XROOTD (Buffered), 570 Series 1, RFIO, 3180 Series 1, File Stager, 503 Series 1, Local , 286 Series 1, XROOTD (NOT Buffered), 1, 196 E+06 Datatransfer in Byte Series 1, XROOTD (Buffered), 3, 827 E+07 Series 1, RFIO, 8, 307 E+07 Series 1, File Stager, 1, 015 E+07 Series 1, Local , 8, 131 E+03 Series 1, Root Read Data Volume, 1, 155 E+06 Series 1, Total Data Volume, 9, 551 E+06 Max Baak & Matthias Schott 11/29