RFTP The File Transfer Protocol Application in 100

RFTP: The File Transfer Protocol Application in 100 Gbps Networks Brookhaven National Laboratory and Stony Brook University

Outline n Project Introduction n Challenges and Requirements n Software System Design n Project Status and Future Work n Testbed Evaluation and Demo Results

Today’s Data-intensive Applications n DOE Leadership Computing Facilities, Data centers, grid and cloud computing, network storage n Explosion of data, and massive data processing n Scalable storage systems n Ultra-high speed network for data transfer: 100 Gbps networks End system connectivity Courtesy of Mellanox

End-to-End 100 G Networking End-to-End Networking at 100 Gbits/s 100 G APPS 100 G APPS FTP 100 Our project and its role 100 G NIC 100 Gbits/s Backbone

Problems Definition and Scope n Reliable Transfer (error checking and recovery) at 40/100 G speed, burden on processing power n Coordinated data transfer flow efficiently traverses file systems and network, data path decomposition • data read-in/write-out: between disks and user memory/ (backend data path) • Transport: Source memory to destination host memory (frontend data path) n Cost-effective end-to-end data transfer (10 x 10 GE v. s. 3*40 GE, v. s. 1 X 100 GE) from sources to sinks

Challenges and Requirements n File System Bottlenecks: how to do file stage-in/out l Kernel/software stacks slow, the same problem as TCP. l Look into software solutions to zero-copy l fopen, sendfile, splice, O_DIRECT, each has some problem or restriction. l Look into hardware RDMA to pull data directly into the user space. n Storage: l Need to support 100 Gbps in/out by disk spindles (also SSD, Flash I/O) l Multiple RAID controllers to accomplish 100 Gbps. 4 l i. e. Multiple files need to be streamed into buffer in parallel. Switch fabric interconnects disk servers and FTP 100 servers. 4 Storage Aggregation from disks into FTP server disk partition.

Opportunities n CPU: Multi-core/Many core/Hybrid processor of CPU/GPGPU for high processing bandwidth per machine n PCI express 3 now is available l IBM x 3750 M 4 with Ivy bridge to be general available June/18/2012 l Mellanox 40 Gbps (true line speed) CE n Flash Drive directly on PCI bus or internal bus l Fusion I/O l Storage Class Memory

NUMA Arrangement Problem NUMA: Non-Uniform Memory Access (NUMA) further complicates the performance problem. Local A NUMA Node Memory To Remote Memory

Our Current Testbed Socket 0 MLNX_0 Socket 1 PCI Bridge Memory Hyper Transport Socket 3 MLNX_1 Socket 2 PCI Bridge Memory System has to be manually configured for performance PCI Bridge Memory

FTP 100 Design Requirements n Such high performance data transfer requires multiple file Read/Write. n Implement the buffer management (stream multiple files into a buffer in the system memory), and provide handshake with the backend file systems n Challenge of synchronization between Read/Write

System Design n Two modes: one based on the conventional TCP/IP stack with zero-copy and kernel bypass, and the other based on RDMA operations over various architectures. System design with RDMA mode

Middleware Layer n The core of our system is the middleware layer, which is responsible for resource management, task scheduling and synchronization, and parallelism of RMDA operations.

A Clean Design of RFTP Workflow n Separate control/data channels, convenient rput/rget user commands, and clean workflow for performance optimization

Extensions to Standard FTP n We implement a set of file transfer commands that are based on RDMA capability, and they support our rput/rget file transfer

Efficient Flow Control n We maximize the read/write of both network and disks by implementing a credit-based flow control An examle of rput:

Why RFTP over Other Tools? n We are aware of other popular data transfer tools and standards. They all attempt to achieve maximum available network bandwidth l Grid. FTP from Globus l UDT, which is implemented in Grid. FTP l Fasp from Aspera. Soft l Fast TCP from Caltech l High. Speed TCP, RFC 3649, and other window-based TCP variants

Advantages n Performance advantages l one-side RDMA semantics, minimum CPU consumption, efficient disk I/O operations, and minimum data-copy. l can easily saturate the network n Flexibility advantages l Two modes: (1) OFED, RDMA operation, and (2) TCP/IP, with kernel zero-copy optimization (sendfile/splice) n Development/deployment advantages l Light-weighted, portability and compatibility (with OFED and minimum upgrade of Linux kernel).

Project Status and Software Features n FTP over RDMA and TCP was implemented. The code was released to Climate 100 project l RDMA l Direct-I/O l Sendfile and splice l Handle multiple files with various sizes, and directories l Parallel transfer of single large files n We continue to improve the software systems n One IPDPS HPGC workshop paper accepted, and an extended version for journal submission

Demo Systems n We held a demo at Super. Computing’ 2011 n The objectives of our demo include, l Validation of the performance, availability, and reliability of RFTP l Demonstration of line-speed bandwidth achieved in networks with 40 Gbps to 100 Gbps

Testbed Evaluation n 100 Gbps WAN testbed (our deployment network) l 100 Gbps IP link between NERSC, ANL, and ORNL l Maximum physical disk bandwidth of 80 Gbps l 10+ pairs of host-to-host connections between NERSC and ANL, and between NERSC and ORNL. n 40 Gbps LAN/MAN testbed (our development network) l 40 Gbps RDMA link in the Long Island area l Up to 40 Gbps SSD bandwidth We configure and tune our system for good performance, e. g. , adjust TCP parameters and variable file sizes,

Example Disk File Transfer Performance Results in DOE ANI Testbed NERSC ORNL ANL, November 17 at SC’ 11

MAN Performance Results n Metropolitan NY testbed n RFTP saturates the physical link with a modest data block size

WAN Performance Results n Disk-to-disk, NERSC to ORNL, 9 host pairs n Memory-to-memory, NERSC to ORNL, 14 host pairs n Fill Up 100 Gbps DOE ANI networks

Future Work n NUMA programming, and assign CPU/Interrupt/Memory/PCI bridge to improve data transfer rate in multi-core systems n Work with ESNet to get Ro. CE work on DOE ANI testbed n Work with ORNL to migrate our FTP system directly on layer 2 Ethernet -based data transport (Common Communication Interface) n Evaluate our software on host with PCI Gen 3 and Mellanox 40 Gbps cards n Emulate long-latency networks, to improve RDMA-base data transfer, and compare with other data transfer applications (Grid. FTP, BBCP, etc) Adding delay and losses Data sink Emulator with netem 3 rd host as a bridge Data source

BACKUP SLIDES

Live Demo n 100 Gbps WAN testbed l 14 pairs of memory-to-memory connections between NERSC and ANL l 9 pairs of disk-to-disk connections between NERSC and ORNL, file sizes are close to 100 G bytes at each host. l Gangalia to monitor aggregate traffic rate out of NERSC n 40 Gbps LAN testbed at BNL l 40 Gbps RDMA Ro. CE link between two hosts at BNL, netqos 03, netqos 04 l Memory-to-memory data transfer

LAN Performance Results n Local RDMA links n RFTP achieves near line- speed throughput