SCLP Segmentoriented Connectionless Protocol for HighPerformance Software Tunneling

The Goal Improving performance of overlay-based virtual networks Our Proposal SCLP • VXLAN (SCLP)

Outline v Backgrounds Ø Network Virtualization Ø Tunneling Protocols Ø L 4 protocol characteristics

Network Virtualization v Multi-tenant Datacenter Networks l Each tenant can have its own virtual

The Overlay-based Approach v NVO 3: Network Virtualization Overlays l RFC 7364, 7365 VM

Tunneling Protocols v L 2 -in-L 3 Tunneling VM Ether VM L 2 -L

Major Tunneling Protocols v VXLAN (RFC 7348) l UDP based l Linux kernel, OVS,

Upcoming Tunneling Protocol v Geneve (RFC draft) l UDP based l TLV option header

Yet Another Tunneling Protocol v STT (Stateless Transport Tunneling, RFC draft) l Pseudo-TCP header

Problems of Existing Protocols v Performance l VXLAN, NVGRE, Geneve Maximum throughput falls to

L 4 Protocol Types v Message-oriented (e. g. UDP) l Packets are independent of

Why is L 4 Protocol Important ? Message-oriented Segment-oriented A VM sends a large

Our Proposal v SCLP: Segment-oriented Connection-less Protocol l L 4 protocol l Segment-oriented l

Protocol Format v Identification: Original payload ID v F: First Segment Flag v Remaining:

How SCLP Works (Tx) Original Payload (3000 bytes) Segmentation (MSS=1468) 'identification': 0 x 12345678

How SCLP Works (Rx) id : size : 0 offset : 0 Payload Buffer

2 -Level Pre-reassembling v 1 st: GRO (Generic Receive Offload) Protocol stack GRO Reassembling

Implementation v VXLAN over SCLP l CVSW component † l Virtual NIC implementation of

Evaluation v Throughput of VM-to-VM communication with Iperf 1. TCP communication 2. Effect of

Evaluation Results (TCP) v TCP communication 26

Evaluation Results (Pre-reassembling) GRO’s reassembling NVE’s reassembling 27

Conclusion v Network virtualization l Overlay-based approach has become popular l VXLAN is a

Slides: 29

Download presentation

SCLP: Segment-oriented Connection-less Protocol for High-Performance Software Tunneling in Datacenter Networks Ryota Kawashima† Shin Muramatsu† Hiroki Nakayama‡ Tsunemasa Hayashi‡ Hiroshi Matsuo† † Nagoya Institute of Technology, ‡ BOSCO Technologies, Inc.

The Goal Improving performance of overlay-based virtual networks Our Proposal SCLP • VXLAN (SCLP) (UDP) • Geneve (SCLP) (UDP) • … 1

Outline v Backgrounds Ø Network Virtualization Ø Tunneling Protocols Ø L 4 protocol characteristics v Proposal Ø SCLP (Segment-oriented Connection-less Protocol) v Evaluation Ø VM-to-VM communication using VXLAN over SCLP 2

Network Virtualization v Multi-tenant Datacenter Networks l Each tenant can have its own virtual networks l Each virtual network shares the physical network resources Virtual networks Tenant 1 VM VM Tenant 2 VM VM Tenant 3 VM VM VM Physical network 3

The Overlay-based Approach v NVO 3: Network Virtualization Overlays l RFC 7364, 7365 VM NVE : Network Virtualization Edge NVE VM VM Virtual Network NVE Virtual Network VM L 3 tunnel Physical network 4

Tunneling Protocols v L 2 -in-L 3 Tunneling VM Ether VM L 2 -L 4 hdrs Virtual switch Ether Physical server Ether L 3 tunnel Ether Virtual switch Physical server 6

Major Tunneling Protocols v VXLAN (RFC 7348) l UDP based l Linux kernel, OVS, VMware NSX, Cisco Nexus 1000 V … Ethernet (Physical) IP (Physical) UDP VXLAN Ethernet (Virtual) Payload FCS v NVGRE (RFC draft) l GRE based (no L 4 protocol) l Microsoft Hyper-V Ethernet (Physical) IP Ethernet NVGRE (Physical) (Virtual) Payload FCS 7

Upcoming Tunneling Protocol v Geneve (RFC draft) l UDP based l TLV option header l H/W segmentation offload (future) Ethernet (Physical) IP (Physical) UDP Geneve Opt. Ethernet (Virtual) Payload FCS 8

Yet Another Tunneling Protocol v STT (Stateless Transport Tunneling, RFC draft) l Pseudo-TCP header Ø Exploiting TSO (TCP Segmentation Offload) feature Ø Semantics of header fields are modified l VMware NSX Ethernet (Physical) IP (Physical) p. TCP STT Ethernet (Virtual) Payload FCS Protocol number is 6 (TCP) "This is a usual TCP packet !" NIC 9

Problems of Existing Protocols v Performance l VXLAN, NVGRE, Geneve Maximum throughput falls to one-half ! v Compatibility l STT Middleboxes can discard STT packets ! 10

L 4 Protocol Types v Message-oriented (e. g. UDP) l Packets are independent of each other v Segment-oriented (e. g. TCP) l Each packet has “byte-level sequence” l Consecutive packets can be reassembled 12

Why is L 4 Protocol Important ? Message-oriented Segment-oriented A VM sends a large Ethernet frame The frame is encapsulated and divided to multiple packets Each packet is decapsulated and forwarded to a destination VM Consecutive packets are reassembled The VM handles lots of frames Each reassembled packet is decapsulated and forwarded to a destination VM The VM handles fewer frames 13

Packet Structure Example (Tx) 14

Packet Structure Example (Rx) 15

Our Proposal v SCLP: Segment-oriented Connection-less Protocol l L 4 protocol l Segment-oriented l Connection-less l Usage: Outer L 4 protocol of existing tunneling protocols Ø e. g. ) VXLAN over SCLP, Geneve over SCLP Ethernet (Physical) IP (Physical) UDP VXLAN Ethernet (Virtual) Payload FCS Ethernet (Physical) IP (Physical) SCLP VXLAN Ethernet (Virtual) Payload FCS 17

Protocol Format v Identification: Original payload ID v F: First Segment Flag v Remaining: Remaining payload size 18

How SCLP Works (Tx) Original Payload (3000 bytes) Segmentation (MSS=1468) 'identification': 0 x 12345678 SCLP Payload 1 (1468 bytes) 'F': 1 'remaining': 1532 SCLP Payload 2 (1468 bytes) 'F': 0 'remaining': 64 SCLP Payload 3 (64 bytes) 'F': 0 'remaining': 0 19

How SCLP Works (Rx) id : size : 0 offset : 0 Payload Buffer (NULL) ① Received payload 1 id : 0 x 12345678 size : 3000 offset : 1468 (id = 0 x 12345678, length = 1468, F = 1, remaining = 1532) 1468 ② Received payload 2 id : 0 x 12345678 size : 3000 offset : 2936 id : 0 x 12345678 size : 3000 offset : 3000 (id = 0 x 12345678, length = 1468, F = 0, remaining = 64) 2936 offset == size – (len + rem) (normal order !) ③ Received payload 3 (id = 0 x 12345678, length = 64, F = 0, remaining = 0) 3000 20

2 -Level Pre-reassembling v 1 st: GRO (Generic Receive Offload) Protocol stack GRO Reassembling NIC driver v 2 nd: NVE’s decapsulation processing VM NVE Protocol stack Reassembling & Decapsulation 21

Implementation v VXLAN over SCLP l CVSW component † l Virtual NIC implementation of NVE VM Applications Protocol stack CVSW v GSO/GRO offloading Encapsulation Decapsulation virtio l Linux kernel module OVS Protocol stack Offload module GSO (SCLP) GRO (SCLP) NIC driver † https: //github. com/sdnnit/cvsw_net Physical server 22

Evaluation v Throughput of VM-to-VM communication with Iperf 1. TCP communication 2. Effect of 2 -level pre-reassembling v Competitors l VXLAN (over UDP) l NVGRE l STT l Geneve (w/o HW offloading) 24

Evaluation Environment 25

Evaluation Results (TCP) v TCP communication 26

Evaluation Results (Pre-reassembling) GRO’s reassembling NVE’s reassembling 27

Conclusion v Network virtualization l Overlay-based approach has become popular l VXLAN is a de-facto tunneling protocol l UDP-based tunneling has performance problems v Proposal: SCLP l Segment-oriented and connection-less L 4 protocol l 2 -level pre-reassembling before decapsulation l STT-comparable performance v Future work l Implementation of OVS-based SCLP l Open source 28