Mas Q RDMA for Virtual Private Cloud Zhiqiang

Mas. Q: RDMA for Virtual Private Cloud Zhiqiang He 1, 2 Dongyang Wang 2, Binzhang Fu 2, Kun Tan 2, Bei Hua 1, Zhi-Li Zhang 3, Kai Zheng 2 University of Science and Technology of China 1 Huawei Technologies Co. , Ltd 2 University of Minnesota 3

VPC needs RDMA for high-performance network Public cloud VPC VM VM … VM • Bandwidth hungry • Latency sensitive • Or both Virtual network • 200 Gbps tput and higher • 2 s RTT under one switch 2

Network virtualization for the traditional TCP/IP network A virtual network VM VM A B 192. 0. 0. 1 192. 0. 0. 2 VM C 192. 0. 0. 3 v. Switch Host 10. 0. 0. 1 VM D 192. 0. 0. 4 v. Switch Host 10. 0. 0. 2 v. Switch is introduced to realize packet-based network virtualization. This illustration is inspired by Slim’s presentation in NSDI 19. 3

Network virtualization for the traditional TCP/IP network A virtual network VM VM B A 192. 0. 0. 1 192. 0. 0. 2 192. 0. 0. 4 data VM C 192. 0. 0. 3 v. Switch Host VM D 192. 0. 0. 4 data v. Switch 10. 0. 0. 1 192. 0. 0. 4 Hostdata 10. 0. 0. 2 v. Switch is introduced to realize packet-based network virtualization. This illustration is inspired by Slim’s presentation in NSDI 19. 4

Problems raised when virtualizing RDMA network TCP/IP RDMA VM Software Where and VM Application data TCP/IP how tostack realize RDMA IPIP data v. IP data Host v. Switch Hardware NIC Application network virtualization? Host v. Switch data RNIC h. IP IPIP data v. IP data RDMA stack IPIP data v. IP data 5

RDMA communication VM: 192. 0. 0. 1 Application (client) 1. QP = setup(192. 0. 0. 2) data 2. QP. post _send() data 3. QP. cleanup() RNIC VM: 192. 0. 0. 2 Application (server) 1. QP = setup(192. 0. 0. 1) data 2. QP. post _recv() 3. QP. cleanup() 6

RDMA communication VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. cleanup() 192. 0. 0. 2 RNIC RDMA stack RDMA driver QPC Cached Dst. IP -> 192. 0. 0. 2 QPC Cached QPC 7

192. 0. 0. 2 RDMA communication VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. cleanup() data RNIC RDMA stack 192. 0. 0. 2 data RDMA driver QPC Dst. IP -> 192. 0. 0. 2 Cached QPC 8

RDMA communication VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. cleanup() 192. 0. 0. 2 RNIC RDMA stack Virtualize RDMA network by controlling the QPCs! RDMA driver QPC Dst. IP -> 192. 0. 0. 2 Cached QPC 9

Mas. Q architecture VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() 192. 0. 0. 2 RNIC RDMA stack Host Mas. Q QPC QPC 10

Mas. Q architecture VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() 192. 0. 0. 2 RNIC RDMA stack Host Mas. Q QPC QPC 11

Mas. Q architecture VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() communication Good RDMA 3. QP. destroy() 192. 0. 0. 2 RNIC RDMA stack performance! Host Good capability to realize network features! Mas. Q QPC QPC 12

192. 0. 0. 2 Three challenges VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() Host Mas. Q How to RNIC provide an stack Ro. CE RDMA device? Network virtualization? QPC QPC security rules? Applying 13

192. 0. 0. 2 Ro. CE device abstraction VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() veth RNIC RDMA stack v. Bond vrdma Host Mas. Q 14

192. 0. 0. 2 Network virtualization VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() Host : 10. 0. 0. 1 10. 0. 0. 2 RNIC RDMA stack vrdma 192. 0. 0. 2 Mas. Q QPC Dst. IP -> 10. 0. 0. 2 15

192. 0. 0. 2 Network virtualization VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() 10. 0. 0. 2 data vrdma RNIC RDMA stack 10. 0. 0. 2 data Host Mas. Q QPC Dst. IP -> 10. 0. 0. 2 Cached QPC 16

192. 0. 0. 2 Applying security rules VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() 10. 0. 0. 2 RNIC RDMA stack vrdma Host: 10. 0. 0. 1 Mas. Q QPC 17

192. 0. 0. 2 Applying security rules VM: 192. 0. 0. 1 Application 1. QP = setup(192. 0. 0. 2) 2. QP. post _send() 3. QP. destroy() 10. 0. 0. 2 RNIC RDMA stack data vrdma Host: 10. 0. 0. 1 Mas. Q QPC State -> error Cached QPC 18

Evaluation Goal: Does Mas. Q provide competitive performance? Four candidates: • Host-RDMA (baseline) • SR-IOV (baseline in VM) • Free. Flow (SOTA software-based solution) • Mas. Q Testbed setup: • Two servers connected with 40 Gbps Ro. CE NICs (Mellanox CX 3 Pro Ro. CE) • Two Intel Xeon E 5 -2690 14 core CPUs, 96 GB RAM • QEMU VMs (Docker containers for Free. Flow) 19

Does Mas. Q provide low latency? Write (2 B) Send (2 B) Write (2 B) Latency ( s) Send (2 B) Mas. Q achieves almost the same latency as SR-IOV or Host-RDMA. 20

Does Mas. Q provide high throughput? Write Throughput (Gbps) Send Message size (B) Mas. Q achieves the same throughput as SR-IOV or Host-RDMA. Mas. Q performs better than Free. Flow for small messages (<8 KB). 21

Does Mas. Q work well in different applications? Graph-500 (HPC) KVS (Web system) Spark-RDMA (Big data) 10 Mas. Q performs well in all applications. 22

Does Mas. Q introduce any overhead? • Takes a slightly longer time in QP setup stage. • About 0. 2 ms (%11) longer compared to SR-IOV (1. 9 ms) in our test case. 23

Conclusion Contributions of this work: • Mas. Q first proposes to virtualize RDMA network in QP setup phase. • Mas. Q further proposes three practical mechanisms to realize features required by VPC. • A prototype of Mas. Q is implemented and the evaluation proves the efficiency. Future work: • Support containers • Support migration for RDMA-capable VMs 24

Thank you! Questions are welcome. zhqhe@mail. ustc. edu. cn 25

Backup slides 26

Rethink RDMA network virtualization architecture Software VM Hardware VM VM Application Host Application RNIC Net virt. NIC Hybrid RDMA stack Special HW RDMA stack Network virt. • Good flexibility • Poor performance • Good performance • Need special HW Application Host Network virt. RNIC RDMA stack • Good performance • Good flexibility 27

Design challenges VM Application Performance challenges • Which operations should be bypassed? Host Mas. Q RNIC RDMA stack Feature challenges • Network virtualization? • Applying security rules? • Quality of service? 28