Net Lord A Scalable Multitenant Network Architecture for

Net. Lord: A Scalable Multi-tenant Network Architecture for Virtualized Cloud Datacenters Jayaram Mudigonda Praveen Yalagandula, Jeff Mogul, Bryan Stiekes, Yanick Pouffary Hewlett-Packard Company © Copyright 2010 Hewlett-Packard Development Company, L. P.

The Goal Build the right network for a cloud datacenter? 2 12 June 2021

The Goal Build the right network for a cloud datacenter? 3 12 June 2021

Cloud Datacenter • Provides Infrastructure as a Service − Shared across multiple tenants • Virtualized − Tenants run Virtual Machines (VMs) 4 12 June 2021

The Right Network • Virtualization + Multi-tenancy − A Virtual Network to each tenant − Flexible abstraction: No restrictions on addressing or protocols • Scale − Tenants, Servers: 10 s of 1000 s, VMs: 100 s of 1000 s − Adequate bandwidth • Inexpensive − Capital: Cheap COTS components − Operational: Ease of management, High bisection BW 5 12 June 2021

The Challenge • Basic COTS switching gear − Limited functionality − Limited resources: • Not enough Forwarding Information Base (FIB) space • COTS Switches + Bisection BW Multipathing • Multipathig and/or Scale More switch resources 6 12 June 2021

The Challenge Assume no switch support and conserve switch resources (FIB) to Simultaneously achieve: Scale, Multi-tenancy, Bisection BW, and Ease of config 7 12 June 2021

State of the Art – Scale • Most prior work is limited by one or more of: − New protocols − Modified control and/or data planes − Preferred topologies − Resources (such as table space) on switches 8 12 June 2021

State of the Art – Multi-tenancy • Traditional VLANs − Single tenant − No L 3 abstraction − Cannot scale beyond 4 K • Commercial: Amazon EC 2 and Microsoft Azure • Most of recent work is on segregation not virtualization • Diverter [WREN 2010] − Carves a SINGLE IP address space among tenants − Restricts: VM placements and IP address assignment − Can be limited by the FIB space in switches 9 12 June 2021

Net. Lord • An encapsulation scheme and A complementary switch configuration Scalable multi-tenancy • Ease of configuration • Order of magnitude reduction in FIB requirements • High bisection BW • 10 12 June 2021

Why Encapsulate? • Unmodified VM packets onto the network − Excessive FIB pressure and no L 3 abstraction • Alternative: Rewrite headers − Cannot assume L 3 header Rewrite only MAC hdr • Rewrite with server MACs Somewhat reduced FIB • Cannot identify the right VM on destination Server 11 12 June 2021

Net. Lord Encapsulation • Two headers: a MAC and an IP • Reduced FIB pressure − Outer Src MAC = MAC of the Src edge switch − Outer Dst MAC = MAC of the Dst edge switch • Correct delivery − Right edge switch: The outer MAC header − Right server: Right port # in the outer dest IP addr − Right VM: Tenant-ID frm outer dest IP + Inner dest MAC • Clean abstraction − No assumptions about VM protocols and/or addressing 12 12 June 2021

Net. Lord Encapsulation VM 1 Tenant 15 Server 1 Hypervisor Net. Lor d Agent ES-MAC 1 P 1 IP Address Format 0 Port# 7 bits VM 2 Tenant 15 Tenant. ID 24 bits Hypervisor Net. Lor d Agent COTS Ethernet VM 1 VM 2 13 12 June 2021 IP-Range-2 ES-MAC 2 P 2 ES-MAC 1 P 1. 0. 0. 15 P 2. 0. 0. 15 Server 2 ES-MAC 1 ES-MAC 2

Switch Configuration • Outer MAC hdr takes pkt to egress edge switch • A switch on MAC Pkt addressed to itself − Strips MAC hdr and forwards based on IP hdr inside − Standard behavior • Correct forwarding − Configure the L 3 forwarding tables right − Make sure to match the server configs 14 12 June 2021

Switch Configuration Server 1 Server 2 Hypervisor 1. *. *. * 1. 0. 0. 1 NLA 15 NLA Prefix Gateway, port 1. *. *. * 1. 0. 0. 1, 1 2. *. *. * 2. 0. 0. 1, 2 … … 48. *. *. * 48. 0. 0. 1, 48 12 June 2021 Server 48 … 2. *. *. * 2. 0. 0. 1 2 1 Hypervisor NLA … 48 48. *. *. * 48. 0. 0. 1

NL-ARP • Builds and maintains the mapping <VM-MAC, Subnet, Tenant-ID> <SWITCH-MAC, PORT> • Query – response. Similar to regular ARP • NL-Agent broadcasts on VM boot-up and migration 16 12 June 2021

Goals Achieved: Abstraction • No assumptions about VM-to-VM pkt contents No restrictions on addressing schemes • No restrictions on L 3 protocols • 17 12 June 2021

Goals Achieved: Scale • 24 -bit tenant ID Millions of tenants • Order of magnitude reduction in FIB requirements − Non-edge switches see only edge-switch MACs − Edge switches see their local server MACs in addition • Small number of L 3 table entries in edge switches − One per downlink 18 12 June 2021

Goals Achieved: Ease of Config Automatic • Local • Same across all switches Easy to debug • Does not change Easy to debug • • Easy per-tenant traffic engineering in the network − Tenant ID readily available in standard IP header 19 12 June 2021

Bisection Bandwidth • Reduced FIB pressure Utilize SPAIN effectively J. Mudigonda, P. Yalagandula, M. Al-Fares, and J. C. Mogul. SPAIN: COTS Data-Center Ethernet for Multipathing over Arbitrary Topologies. In Proc. USENIX NSDI, 2010. 20 12 June 2021

SPAIN in One Slide • Multipath forwarding in container/pod datacenters − Few thousand servers (or unique MAC addresses) − COTS Ethernet (with VLAN support) − Arbitrary topologies • A central, off-line manager: − − • Discover topology Compute paths; exploit redundancy in the topology Merge paths into a minimal set of trees Install trees as VLANs into switches End-host (NIC driver): − Download VLAN maps from the manager − Spread flows over VLANs (i. e. , paths) 21 12 June 2021

SPAIN & Extreme Scalability • Critical resource: Forwarding table entries (FIB) − Each MAC address appears on multiple VLANs − Worst case FIB = #VLANS × #MACs − Overflowing FIB Floods and poor goodput • 64 K Entries only a few 1000 unique MACs • Net. Lord’s ability to hide MACs helps greatly 22 12 June 2021

Putting It All Together Server 1 VM 1 Server 2 VM 1 VM 2 Hypervisor Net. Lor d. VM 1 VM 2 Agent P 1. *. *. * IP 1 I P 2 mac. A ma c. B Net. Lor d VM 1 VM 2 Agent VM 1 VM 2 COTS Ethernet 12 June 2021 IP 1 I P 2 mac. B mac. A 23 IP 1 I P 2. *. *. * P 2 mac. A ma c. B

Evaluation • Analytical arguments: − Max VMs : ~650 K using 48 -port 64 K FIB switches − NL-ARP overheads: < 1% of link BW, ~0. 5% of Server memory • Prototype + VM emulation − NLA, encaping overheads are ignorable − Substantial goodput improvements 24 12 June 2021

Experimental Setup: Topology • 25 74 Servers in a 2 -level fat-tree topology 12 June 2021

Prototype, VM Emulation User-level Process • NLA Kernel module … • VM Emulator Protocol Stack − A thin module above NLA − Exports a virtual device − Re-writes MAC addresses VM Emulator Net. Lord Agent NIC Driver • Unique pair for each TCP cxn Physical NIC • Network 26 12 June 2021 Up to 3 K VMs / Servers − 200 K VMs in all

Experimental Setup: Workload • Parallel shuffles − Emulating shuffle-phase of Map-Reduce jobs − Each shuffle: 74 mappers & 74 reducers, one per server − Each mapper transfers 10 MB of data to all reducers 27 12 June 2021

Goodput 25 Traditional SPAIN Goodput (in Gbps) 20 Net. Lord 15 10 5 0 74 3. 7 K 7. 4 K 14. 8 K 37 K Number of VMs 74 K 148 K 222 K

Flooded Packets (in Millions) Floods Number of VMs

Summary Net. Lord combines simple existing primitives in a novel fashion to achieve several out-sized benefits of practical importance: Scale • Multi-tenancy • Ease-of-use • Bisection BW • 30 12 June 2021

• 31 BACKUP 12 June 2021

Putting It All Together Server 1 VM 1 Server 2 VM 1 VM 2 Hypervisor Net. Lor d. VM 1 VM 2 Agent P 1. *. *. * IP 1 I P 2 mac. A ma c. B Net. Lor d VM 1 VM 2 Agent VM 1 VM 2 MAC-A COTS Ethernet 32 12 June 2021 IP 1 I P 2. *. *. * P 2 mac. A ma c. B