Datacenter Network Challenges customers face Increase agility I

Datacenter Network

Challenges customers face Increase agility “I need to onboard workloads with complex policies across my own datacenter and/or other clouds in days – not weeks – to remain competitive. ” Enhance security “I must be able to instantaneously react to evolving threats and stop an attack from spreading. ” Reduce costs “I need to reduce the number of operator interventions and efficiently meet network growth demands. Current practices just won’t scale. ”

Agility Security Costs

“We had a large power outage… UPS did not last as long as expected… connectivity is gone. ”

Expose Actionable Error Messages to the Admin

NIC 1 NIC 2 NIC 1 NIC 2 AD DNS SQL

VMM Logical Networks NIC 1 NIC 2 Mgmt SQL SCOM

Network Controller Managed Logical Networks NIC 1 NIC 2 Mgmt NC Host Agent NC 2 Mgmt NC 3 NC Host Agent Mgmt SQL NC Host Agent SCOM

Transit NIC 1 NIC 2 Mgmt Network Controller Managed Logical Networks MUX Mgmt HNV 1 Transit Mgmt NC 2 Mgmt MUX Mgmt HNV 2 Transit NC Host Agent SLB Host Agent RDMA 2 Mgmt NC 3 Mgmt MUX Mgmt HNV 3 Transit NC Host Agent SLB Host Agent SQL NC Host Agent SLB Host Agent NIC 1 NIC 2 Mgmt SCOM

Transit NIC 1 NIC 2 Mgmt Network Controller Managed Logical Networks NIC 1 NIC 2 Mgmt NC Host Agent Mgmt SLB Host Agent NC Host Agent SLB Host Agent NIC 1 NIC 2 Mgmt NC Host Agent SLB Host Agent SQL NC Host Agent SLB Host Agent SCOM

Transit NIC 1 NIC 2 Mgmt Network Controller Managed Logical Networks NIC 1 NIC 2 Mgmt NC Host Agent Mgmt SLB Host Agent NC Host Agent SLB Host Agent NIC 1 NIC 2 Mgmt NC Host Agent SLB Host Agent SQL NC Host Agent SLB Host Agent SCOM

Network Controller Managed Logical Networks Network Controller Managed Virtual Networks HNV PA VFP HNV PA VFP

Network Controller Managed Virtual Networks Tra ns it VFP MUX Mgmt HNV 1 Transit VFP MUX Mgmt HNV 2 Transit MUX Mgmt HNV 3 Transit VFP

Network Controller Managed Virtual Networks Tra ns it VFP MUX Mgmt HNV 1 Transit VFP MUX Mgmt HNV 2 Transit MUX Mgmt HNV 3 Transit VFP

Network Controller Managed Logical Networks Network Controller Managed Virtual Networks HNV PA VFP HNV PA VFP

What resources do I need to monitor? What resources can I monitor? Need Actionable Information *Actually, what we really want is to be warned about an impending problem before it occurs… Not there yet

Monitoring Packs (MP)

These resources have implemented configuration state in the Network Controller § Servers § Network Interfaces § Virtual Networks § Load Balancers § Gateways § Access Control Lists

Alerts View Stamp View Resource View Change Resolution State Forward Close Inspect Alert Description and Status Product Knowledge lists possible causes and resolutions

https: //technet. microsoft. com/en-us/library/mt 490472. aspx https: //technet. microsoft. com/en-us/library/mt 490456. aspx https: //github. com/Microsoft/SDN/tree/master/Diagnostics Troubleshooting Guidance on Tech. Net: https: //technet. microsoft. com/en-us/windows-server-docs/networking/sdn/troubleshoot-windows-server 2016 -software-defined-networking-stack

Hardware accelerators Features are in OS software No, but many do No, but many are

Address checksum offload group Data Center Bridging (DCB) Interrupt moderation (IM) IPsec Task Offload (IPsec. TO) Jumbo frame Large Send Offload (LSO) NVGRE Task Offload Receive Side Coalescing (RSC) Receive Side Scaling (RSS) Single Root I/O Virtualization (SR-IOV) Remote Direct Access Memory (RDMA) TCP Chimney Offload Virtual LAN (VLAN) support Virtual Machine Multi-Queue (VMMQ) Virtual Machine Queues (VMQ) Vx. LAN Task Offload Feature-driven Offload only Sometimes feature driven

Overview Affects interactions with other features Not all hardware features provide the results we hope for E. g. , TCP Chimney Not all vendors are created equal No, we don’t publish competing performance numbers

WS 2012 R 2 Data Plane features Extensible switch (Cisco, NEC, etc. ) vm. Qo. S ACLs and extended ACLs Hyper-V Virtual Networking (NV-GRE)

WS 2012 R 2 Data Plane Offloads

WS 2016 - Moving to the future 1. 2. One carries the WS 2012 R 2 stack forward with little change; One has the new SDN Switch Extension, high-performance VM support, and cost-reduction capabilities that are shared with Azure This is the SDNv 2 feature set N SD h t i w 16 WS 2012 R 2 s e r u t a v 2 fe WS 2016 without SDNv 2 features

WS 2016 - Moving to the future WS 2016 base features are the WS 2012 R 2 features plus: • Software v. RSS extended to host v. NICs • Small enhancements to NIC Teaming • Switch Embedded Teaming • Virtual Machine Multi. Queues (VMMQ) • Converged NIC (RDMA to the host v. NIC) WS 2016 SDNv 2 feature set is built on the SDN Switch Extension • Integrated ACLs (Replaces and extends Hyper-V switch ACLs) • HNVv 2 TO (Replaces HNVv 1, adds Vx. LAN) • SDN Qo. S (Replaces and extends vm. Qo. S)

Managing the network stack Managed by Power. Shell or SCVMM NIC Teaming managed by Power. Shell or lbfoadmin. exe (linked from Server Manager) or SCVMM Fabric managed by SCVMM Physical and Virtual networks managed by Network Controller (NC) No UI for Tenant management or Switch Extension features

How do the new features work? (and why do we have them)? SDN Switch Extension a. k. a. , Virtual Filtering Platform (VFP) Includes: • Integrated ACLs • HNVv 2 (Vx. LAN, NV-GREv 2) • SDN Qo. S VMMQ Converged NIC Client RDMA SET

v. NIC VM VM vm. NIC VM Switch VFP ACLs, Metering, Security VNET SLB (NAT)

HNVv 2 – Vx. LAN, NV-GRE But we also do NV-GRE for those who like that option A semi-hidden feature automatically adjusts the MTU on the wire to accommodate the encapsulation overhead Better performance than splitting packets due to length of encapsulation

Virtual Machine Multi. Queue (VMMQ) Each queue affinitized to a single CPU Each CPU capable of 3. 5 -5 Gbps of packet processing One queue/one CPU for default queue processing Each queue still affinitized to a single CPU Each VM can have a different number of queues Default queue becomes a set of queues interrupting a set of CPUs Toeplitz hash spreading (RSS) used to spread traffic between queues for the same VM

Queue 0 Packet Arrival Queue 1 Queue 2 Queue 3 Queue … Queue N Calculate Toeplitz hash Distribute to queue, interrupt processor modulo # of queues

VM 1 Packet Arrival VM 2 VM 3 Select dest MAC+VLAN, Compare to filters VM … VM N Default Distribute to queue, interrupt processor

Default Port 1 Packet Arrival Port 2 Port 3 Port 4 Port … Port N Embedded NIC Switch with forwarding table Fwd table MAC+VLAN Port # Else (Default) Port #

Queue 0 Queue 1 Queue … Queue N vm. Bus subchannels vm. NIC vm. Bus P 1 P 2 Hyper-V switch P 3 P 4 P… Pn de Mux VMQ in physical NIC

Queue 0 Queue 1 Queue … Queue N vm. Bus subchannels vm. NIC vm. Bus P 1 P 2 Hyper-V switch P 3 P 4 P… Pn de Mux VMQ in physical NIC

Topl. Hash Port 1 Packet Arrival Topl. Hash Port 2 Topl. Hash Port 3 Topl. Hash Port 4 Topl. Hash Port … Topl. Hash Port N Embedded NIC Switch with forwarding table Fwd table MAC+VLAN Port # Else (Default) Port #

Cavium QLogic® Fast. Lin. Q™ Ethernet NICs 10 Gb. E Intelligent Adapter 10 Gb. E CNA Direct Attach Copper SR Optical Direct Attach Copper Active Optic Cables 10 GBASE-T DAC and SR Direct Attach Copper SR Optical QL 45212 HLCU QLE 340 X QLE 840 X 1 √ 2 QL 45412 HLCU Ports QL 45611 HLCU 1 2 √ √ © 2016 Cavium, Inc. – Confidential and Proprietary Information 1 2 √ √ Ports 25 Gb. E Intelligent Adapter Ports 40 Gb. E Intelligent Adapter Ports 100 Gb. E Intelligent Adapter 1 2 √ √

• Ro. CE, Ro. CEv 2, i. WARP - Leverage and Scale Windows Server 2016 SMB Direct, Live Migration over RDMA • Converged NIC – Concurrent RDMA and Hyper-V support with Switch Embedded Teaming (SET) • VXLAN and NVGRE Offload: Optimize Hyper-V Network Virtualization (HNV 2) to scale the hybrid Cloud • i. SCSI and FCo. E Offloads: High performance storage without CPU burden Universal RDMA Offloads New Ethernet Speeds: 10/25/40/50/100 Gb. E Connectivity options enable highly available and scalable software-defined storage (SDS) Scalable RDMA: Accelerate Storage Spaces and Storage Spaces Direct (S 2 D) Scalable Hyper. Convergence Virtualization Optimized • VMMQ: Accelerate Virtual Machine Networking Traffic • NIC Switch: Offload Virtual Switching w/o SR-IOV • SR-IOV – Direct access from VMs to NIC

Converged NIC Reduce by half the number of required NIC ports in the host Reduce by half the number of required switch ports in the rack

Converged NIC – the way we were VM Storage Host partition Live Migration Mgmt Other Stuff SMB Multichannel & SMB Direct VM VM VM vm. NIC Live Migration Management / Cluster Other Stuff SMB TCP/IP RDMA SMB NIC DCB Host partition VM VM VM vm. NIC Hyper-V Switch (SDN) NIC Team With embedded teaming NIC NIC DCB Windows Server 2012 R 2 NIC DCB Windows Server 2016

Converged NIC – the new way VM Storage Host partition Live Migration SMB Mgmt VM VM VM vm. NIC SMB Management/ Cluster Other Stuff TCP/IP SMB Multichannel & SMB Direct Live Migration RDMA Other Stuff NIC DCB Host partition VM VM VM vm. NIC Hyper-V Switch (SDN) NIC Team With embedded teaming NIC NIC DCB Windows Server 2012 R 2 NIC DCB Windows Server 2016

Converged NIC – Bandwidth management Bandwidth managed per VM SDN Qo. S Normal VM RDMA traffic TC=0 3 Special, e. g. , cluster traffic 7 Bandwidth on the wire managed through DCB

Client RDMA

Leading Unified Wire™ Architecture Converged Network Architecture with all-in-one Adapter and Software Networking ✓ 10/25/40/50/100 Gb. E speeds ✓ Full Protocol Offload ✓ Data Center Bridging ✓ Hardware Firewall ✓ Wire Analytics ✓ DPDK/Netmap Storage Virtualization ✓ NVMe/Fabrics ✓ SMB Direct ✓ i. SCSI and FCo. E with T 10 -DIX ✓ i. SER and NFS over RDMA ✓ p. NFS (NFS 4. 1) and Lustre ✓ NAS Offload ✓ Diskless boot ✓ Replication and failover ✓ Hypervisor offload ✓ SR-IOV with embedded VEB ✓ VEPA, VN-TAGs ✓ VXLAN/NVGRE ✓ NFV and SDN ✓ Open. Stack storage ✓ Hadoop RDMA Media Streaming HPC ✓ Traffic Management ✓ Video segmentation Offload ✓ Large stream capacity ✓ i. WARP RDMA over Ethernet ✓ GPUDirect RDMA ✓ Lustre RDMA ✓ p. NFS (NFS 4. 1) ✓ Open. MPI & MVAPICH HFT ✓ ✓ Wire. Direct Technology Ultra low latency Highest messages/sec Wire rate classification Encryption Chelsio Unified Adapter 10/25/40/50/100 Gb. E Single Qualification – Single SKU, Single FW Concurrent Multi-Protocol Operation ✓ IPsec/TLS/SSL Offload ✓ SMB Direct AES Offload

Chelsio’s T 6 ASIC Provides • • • Two 10/25/40/50/100 Gb Ports On-board Offload Crypto Engine Ultra low latency and high IOPS PCIe Gen 3 x 16 And much more. . 6

6

SDN Qo. S Outbound reservations Outbound limits Inbound limits (minimum guaranteed bandwidth) (maximum permitted bandwidth)

Switch Embedded Teaming (SET) But it isn’t compatible with the SDN switch extension Focused on the needs of the SDN Extension and Converged NIC

Switch Embedded Teaming (SET) Switch independent teaming Dynamic or Hyper. VPort modes of load distribution RDMA/DCB aware SR-IOV teaming Teams of up to 8 ports All team members must be identical make/model/driver/features No LACP No Active/Passive teaming

Regulations UTC Time Accuracy UTC Traceability

Windows Server 2016 Time Improvements Latest Generation NTP time synchronization • Better statistical processing • Improved error correction • 500μs skew/200μs RMS Distribution of Precision Time in VM Guests • New version of VMIC protocol • Transparent Stratum • 90μs skew/100μs RMS under load Better Monitoring • New Performance Monitor Counters • Determine Accuracy • Remote Monitoring

Windows Server 2016 Time Topology GPS Stratum 0 PC I e Hex boxes represent accuracy compared to master clock 5 us skew 10 us RMS WS 2016 Time Master 500 us skew 250 us RMS NTP N Hops N > 0, N Affects Accuracy VM can be DCs for the Host Partition OS, this can cause loops 50 us skew 10 us RMS NTP WS 2016 DC OR Non Domain Windows Client WS 2016 Domain Member/Hyper-V Host MS-NTP Host Partition WS 1 -VM VMIC BUS 500 us skew 250 us RMS WS*-VM *nix 1 -VM 520 us skew 270 us RMS Hypervisor is just another VM; VMs get time sliced.

1 ms Accuracy with Windows 2016 1 millisecond regulatory requirement

www. microsoft. com/itprocareercenter www. microsoft. com/itprocloudessentials www. microsoft. com/mechanics https: //techcommunity. microsoft. com

http: //myignite. microsoft. com https: //aka. ms/ignite. mobileapp

Explore Windows Server 2016 Software Defined Datacenter