Marrying Open Stack and BareMetal Cloud Pete Lumbis

Our Speakers Pete Lumbis Technical Marketing Engineer Cumulus Networks David Iles Senior Director –

Multi-Tenant Networks § Single tenant private cloud is easy § Multi-tenant is more challenging § Single ownership § Single security policy § Possible competitor tenants (mixing these is real bad) § Shared infrastructure § Multiple security policies © 2018 Mellanox Technologies | Public 3

Multi-Tenant Networks § VLANs § Isolate traffic with network tags (802. 1 q) over shared physical links § Only Layer 3 Routers can cross VLANs § ML 2 can provision VLANs on compute or physical switches ML 2 VLAN Provisioning © 2018 Mellanox Technologies | Public 4

VLANs Are Not Your Friend § Pros: § Solve multi tenancy networking © 2018

VLANs Are Not Your Friend § Pros: § Solve multi tenancy networking § Cons: § Single network path § Low scalability § Scale-Up networking, not scale-out § Limited VLAN range (4096 at best) © 2018 Mellanox Technologies | Public 7

VLANs Are Not Your Friend § Pros: § Solve multi tenancy networking § Cons: § Single network path § Low scalability § Scale-Up networking, not scale-out § Limited VLAN range (4096 at best) § Large blast Radius Impacted Server NIC Failure © 2018 Mellanox Technologies | Public 8

Beyond VLANs: Vx. LAN § Same L 2 extension and isolation as VLANs (Vx. LAN ID) § Operate over L 3 network § Small blast radius § Enables scale-out networking § Extremely high bandwidth network § Smaller ML 2 Footprint ML 2 Vx. LAN Provisioning © 2018 Mellanox Technologies | Public 9

Compute Based Vx. LAN § Vx. LAN from compute to compute § No Vx. LAN based switches § BGP to the server for simplicity § Same config, troubleshooting for both network and compute § No Layer 2 § BGP advertises compute endpoint (VTEP) § ML 2 provisions Vx. LANs per tenant ML 2 Vx. LAN Provisioning © 2018 Mellanox Technologies | Public 10

Compute Based Vx. LAN – The Ugly § Vx. LAN capable NICs required § Most servers have them, but not all. § Performance hit without NIC offload § BGP to the server is scary § It’s not really, but not everyone likes the idea § Network doesn’t want server folks touching their network § Server folks don’t want to learn BGP § Complicated solution for smaller deployments § Difficult for Ironic (bare metal) © 2018 Mellanox Technologies | Public 11

Network Design Recap § VLAN Based § Bad, don’t do it § Even at a few racks § Vx. LAN Compute + Network Based § Great, but still requires network ML 2 § Loss of a switch == loss of a lot of Neutron state § Still requires Vx. LAN NICs § Vx. LAN Compute Only § Best solution if you are okay with BGP on servers § Still requires Vx. LAN NICs § Not the easiest for Ironic © 2018 Mellanox Technologies | Public 12

Network Design Needs § Stability § It isn’t cloud if one host brings down the others § Simplicity § I don’t need the Avengers to run the infrastructure § Flexibility § Works for bare metal and virtual workloads § Scalability § Maybe not 1000 s but 100 s of tenants can be required § Affordable § Vx. LAN NICs may not be an option © 2018 Mellanox Technologies | Public 13

The Answer… § VLAN based compute § Wait…. § + Vx. LAN based network § But it’s not Hierarchical Port Binding § HPB manages both Vx. LAN and VLANs § ML 2 only drives compute VLANs § Network is pre-provisioned § Offload Vx. LAN to the network § No need for special NICs § Localized VLANs easily and safely scales to 100 s of tenants § Larger scale requires more complex solutions © 2018 Mellanox Technologies | Public 14

VLAN + Vx. LAN § Vx. LANs pre-configured between all switches § Compute facing network ports have VLAN trunks configured § ML 2 provisions VLANs on compute nodes as needed § Every VLAN is mapped to an identical Vx. LAN Tunnel (VNI) ML 2 VLAN Provisioning Vx. LANs VLANs © 2018 Mellanox Technologies | Public 15

Vx. LAN Without a Controller? § Switches need Vx. LAN knowledge § WHO: else is a Vx. LAN end point (VTEP) § WHAT: Vx. LAN tunnels exist on each VTEP § WHERE: do MAC addresses live § Option 1: Vx. LAN Controllers § Openstack ML 2 § Open. Contrail/Tungsten Fabric § Open Daylight § Option 2: EVPN § Switches exchange data without a controller § Relies on extensions to BGP § Multi-vendor (Cumulus, Cisco, Arista, Juniper) Vx. LAN Information Controller Vx. LAN Information © 2018 Mellanox Technologies | Public 16

EVPN: A Closer Look A B © 2018 Mellanox Technologies | Public 17

EVPN: A Closer Look § Switch build BGP relationships BGP A B © 2018

EVPN: A Closer Look § Switch build BGP relationships § Switch learns MAC addresses from servers § Just like a normal switch I know about the MAC for server B! I know about the MAC for server A! A B © 2018 Mellanox Technologies | Public 19

EVPN: A Closer Look § Switch build BGP relationships § Switch learns MAC addresses from servers § Just like a normal switch § MAC information exchanged via BGP Come to reach MAC A A Come to reach MAC B B © 2018 Mellanox Technologies | Public 20

EVPN: A Closer Look § Switch build BGP relationships § Switch learns MAC addresses from servers § Just like a normal switch § MAC information exchanged via BGP § Data sent via Vx. LAN based on BGP information From: Switch A To: Switch B From: Server A To: Server B (Vx. LAN) A B © 2018 Mellanox Technologies | Public 21

EVPN: A Closer Look § Switch build BGP relationships § Switch learns MAC addresses from servers § Just like a normal switch § MAC information exchanged via BGP § Data sent via Vx. LAN based on BGP information § Since BGP is used, no shared failures like L 2 A B © 2018 Mellanox Technologies | Public 22

Ba. GPipe – A Potential Future § Neutron work with BGP called Ba. GPipe § Two Goals: § Inter-DC VPNs § Layer 3 to the compute node § Can be done today with Free Range Routing + Neutron Networking § What was described earlier § Ba. GPipe would have Neutron control BGP + Vx. LAN § Today’s solution Neutron only controls Vx. LAN § Nearly identical to EVPN on the server § Extremely early days, but value is clear § Early for Ba. GPipe as well © 2018 Mellanox Technologies | Public 23

How do we make it work with next gen workloads? Machine Learning and NVME Fabrics § Machine Learning Applications § Next Generation Storage: § All Flash § PCIe attached NVME drives § RDMA over Ethernet – Ro. CE § GPU accelerated § PCIe attached GPU’s § RDMA over Ethernet - Ro. CE § Both must run over an Overlay network § Ro. CE + VXLAN ML 2 Vx. LAN Provisioning © 2018 Mellanox Technologies | Public 24

EVPN Gotcha’s § License-free Features: BGP, VXLAN, ZTP, EVPN § VXLAN Routing in Hardware § No Loopback cables § VTEP Scale § Many switches max out at 128 VTEPs § ROCE over VXLAN § NVME over Fabric § Machine Learning © 2018 Mellanox Technologies | Public 25