LowLatency Communication Workshop Lecture 2 Alex Margolin Winter

Low-Latency Communication Workshop Lecture #2 Alex Margolin Winter 2017 1

Lecture Outline 1. Memory registration & ODP Designing MPI library with on-demand paging (ODP) of infiniband: challenges and benefits, Li et el. , SC 2016 2. IB Opcodes: SEND, RDMA READ/WRITE, atomics 3. Protocols: Eager vs. Rendezvous RDMA Read Based Rendezvous Protocol for MPI over Infini. Band: Design Alternatives and Benefits, Sur et el. , PPo. PP 2006 4. Advanced Offloading: Core. Direct Peer. Direct GPUDirect 2

Part 1: Memory Registration 3

Multiplexing & Demultiplexing • Multiplexing: Fit traffic streams on a single channel Occurs at most levels of IP stack Protocols specify with headers/encapsulation Operating Systems also have to schedule messages Which waiting packet do I send now? Examples: IP, IPX, Apple. Talk, etc. over Ethernet TCP, UDP, ICMP, etc. over IP Multiple TCP connections (ports) • De-multiplexing: Directing a packet from a channel Lowest level: (physical!) network interface Highest level: socket (for Unix/Windows. . . ) Traditional approach: demux at multiple layers IP, TCP, etc. Extreme case: run each layer in own thread/process 4

Multiplexing – User-level networks • Basically, Requests in Send/Recv-Queueus point to buffers in the user-space • Multiplexing is not really in software (kernel is not involved) Unlike TCP we typically have significantly less QPs… A Request is posted on the QP, executed FIFO Multiplexing among the QPs is done on the NIC • De-multiplexing is also easy: An incoming packet on the host would have a QPN field – which identifies the QP 5

TCP memory model • In Unix (and most systems): Top half: called from user space (sys-calls, etc. ) Per-process stack, synchronous traps, etc. Bottom half: hardware and software interrupts Dedicated stack, asynchronous w. r. t. top half, etc. • In Linux Top half: hardware interrupts Bottom half: software interrupts 6

OS Reminder: Memory Access • CPU Uses MMU to access the Main memory, other devices use IOMMU • Direct Memory Access (DMA) is a device access to memory directly Access is over a “bus”, such as PCI • The OS manages a “virtual memory” space for the process Some “mapped” to pages main memory, some mapped to HD (swap) Some may be mapped to other devices… 7

Memory Registration • Memory Registration is a mechanism that allows an application to describe a set of virtual or physical contiguous memory locations to the NIC as a virtually contiguous buffer using a Virtual Addresses. The registration process pins the memory pages (to prevent the pages from being swapped out and to keep physical <-> virtual mapping) During the registration, the OS checks the permissions of the registered block The registration process writes to the NIC the translation table from virtual addresses to physical addresses. In Verbs, you use ib_reg_mr() and ib_dereg_mr() to register and release memory. • The result of memory registration is a Memory Region (MR) MR has permission (Local Write, Remote Read, Remote Write, Atom, Bind) Every MR has a remote and a local key (r_key, l_key) which are used in the WR Same memory buffer can be registered several times (with a different access permissions) and every registration results with a different keys • Memory registration is a costly call – try to minimize call in the “fast-path”! 8 8

9 Region Registration & Local Access Host Memory Consumer 1. Register (VA, length) Consumer’s Virtual Address Space Data Buffer 3. Return (L_Key) 5. Access Data (PAs) 4. Submit WR (VA, length, L_Key) Channel Interface Software 2. Program NIC HW (VA, length, PAs, L_Key) NIC 9

10 Region Registration & Remote Access Host Memory Data Buffer Consumer 1. Register (VA, length, remote access) 3. Return (L_Key, R_Key) Channel Interface Software 10 2. Program NIC HW (VA, length, PAs, L_Key, R_Key) 4. Consumer provides Data Buffer info (VA, length, R_Key) to Remote Agent 6. Access Data (PAs) NIC 5. RDMA op (VA, length, R_Key) Remote Agent

Example: This is how I send int foo(struct ibv_pd *pd, struct ibv_qp *qp) { struct ibv_mr *mr; struct ibv_sge sge; struct ibv_send_wr wr, *bad_wr; int length = 16384; char *buf; buf = malloc(length); mr = ibv_reg_mr(pd, buf, length, IBV_ACCESS_LOCAL_WRITE); sge. addr = (uintptr_t)buf; sge. length = length; sge. lkey = mr->lkey; wr->wr_id = (uintptr_t)buf; wr->sg_list = &sge; wr->num_sge = 1; wr->opcode = IBV_WR_SEND; wr->send_flags = IBV_SEND_SIGNALED; } return ibv_post_send(qp, &wr, &bad_wr); 11

Example: This is how I receive int foo(struct ibv_pd *pd, struct ibv_qp *qp) { struct ibv_mr *mr; struct ibv_sge sge; struct ibv_recv_wr wr, *bad_wr; int length = 16384; char *buf; buf = malloc(length); mr = ibv_reg_mr(pd, buf, length, IBV_ACCESS_LOCAL_WRITE); sge. addr = (uintptr_t)buf; sge. length = length; sge. lkey = mr->lkey; wr->wr_id = (uintptr_t)buf; wr->sg_list = &sge; wr->num_sge = 1; } return ibv_post_recv(qp, &wr, &bad_wr); 12

On Demand Paging (ODP) • Communication buffers are no longer pinned Paged in when the NIC needs them Swapped out when reclaimed by the OS • How page fault is handled: Get pages, map to DMA Update mapping, resume communication • Pre-fetch: a new operation to pre-fetch pages Warm up communication buffers No guarantee that pages remain resident 13

ODP performance • Remember OS bypass? Involving the kernel is painful: • As long as the pages are “warm” – there’s no effect: • So, what do we do? Short message buffers may be left pinned (still causes 11 x reduction on pinned memory in some apps), long ones need pre-fetch. 14

Part 2: IB Ops & protocols 15

RDMA opcodes: Send • The responder Post Receive Requests (before data is received) • The requester Post Send Request Only data is sent over the wire • ACK is sent only in reliable transport types Requester Responder sync Post SR data ACK Poll CQ Post RR Poll CQ 16

RDMA opcodes: RDMA Write • The requester Post Send Request Data and remote memory attributes are sent Responder is passive Immediate data can be used to consume RRs at the responder side • ACK is sent only in reliable transport types Requester Post SR Responder data + addr + rkey ACK Poll CQ 17

RDMA opcodes: RDMA Read • The requester Post Send Request Data and remote memory attributes are sent Responder is passive • Data is sent from the responder Available only in reliable transport types Requester Post SR Responder addr + rkey data Poll CQ 18

RDMA opcodes: Atomic • The requester Post Send Request Data and remote memory attributes are sent Responder is passive • Original data is sent from the responder Read-modify-write is performed in responder’s memory Available only in reliable transport types Requester Post SR Responder data + addr + rkey data Poll CQ 19

The “Eager” Protocol • On the Responder – Post some receive buffers (no opcode needed) • On the Requestor – Send to a remote QP (by QPN), using the SEND opcode Sends are typically “SIGNALED”: we want a work completion for it when it’s done Optional: “Completion moderation”, meaning not every send is “signaled” we’ll talk more about it under “Flow control” • This protocol has minimal startup overheads and is used to implement low latency message passing for smaller messages. • The down-side: memcpy! (not “zero-copy”) • What happens if the Responder is not ready? Completion with error! status is not WC_SUCCESS 20

The “Rendezvous” Protocol • Rendezvous is fancy word for “meeting” • On the Requestor – “Eager-Send” notification (just size and location info) • On the Responder – wait for user OR allocate space, then call RDMA_READ OR send back local info (address + rkey), and requestor calls RDMA_WRITE • This protocol is used for transferring large messages when the sender is not sure whether the receiver actually has the buffer space to hold the entire message. • This protocol 21

RDMA_READ vs. RDMA_WRITE • Minimizing control flow RDMA Read does not require a reply to RNDZ_START. • Limited Outstanding RDMA Reads Fixed number of outstanding RDMA reads on any QP. • Issuing FIN Message We have to wait for the RDMA Read completion, before we send back FIN. • Control messages help ODP Use control message arrival to call pre-fetch and avoid page faults. 22

Part 5: Advanced Offloading 23

CORE-Direct Technology • Adapter-based offloading, with these operation types, across QPs: wait - does not complete until a specified number of completions on the listed completion queue complete. receive-enable - operation makes pre-posted operations in the specified receive queue eligible for execution. send-enable - operations does the same for the send queue. Capable of performing 64 -bit integer and floating point data reductions and logical operations, and sending the result of such operations. Ideal System noise CORE-Direct (Offload) CORE-Direct - Asynchronous

Peer. Direct Technology • Based on Peer-to-Peer capability of PCIe • Support for any PCIe peer which can provide access to its memory NVIDIA GPU, XEON PHI, AMD, custom FPGA Peer. Direct RDMA PCIe System Memory GPU Memory PCIe CPU System Memory GPU Memory RDMA Peer. Direct GPU

GPU-Infini. Band Bottleneck • GPU communications uses “pinned” buffers for data movement A section in the host memory that is dedicated for the GPU Allows optimizations such as write-combining and overlapping GPU computation and data transfer for best performance • Infini. Band uses it’s own registered memory System Memory CPU Chip set GPU Infini. Band GPU Memory

Evolution of GPUDirect RDMA Before GPUDirect • Network and third-party device drivers, did not share buffers, and needed to make a redundant copy in host memory. With GPUDirect Shared Host Memory Pages • The network and GPU can share “pinned” (page-locked) buffers, eliminating the need to make a redundant copy in host memory.

GPUDirect (3. 0) RDMA • Eliminates CPU bandwidth and latency bottlenecks • Uses remote direct memory access (RDMA) transfers between GPUs • Resulting in significantly improved MPISend. Recv() efficiency between GPUs in remote nodes

Lower is Better 8. 6 X 10 x 2. 37 usec Source: Prof. DK Panda GPU-GPU Internode MPI Latency GPU-GPU Internode MPI Bandwidth Higher is Better GPUDirect effects on latency & BW

Questions? 30