WSV 321 Windows Server 2012 NIC Teaming and

WSV 321 Windows Server 2012 NIC Teaming and Multichannel Solutions Rick Claus Sr. Technical Evangelist @Rickster. CDN http: //Regular. ITGuy. com

NIC Teaming

Team Interfaces, Team NICs, or t. NICs Team members --or-Network Adapters

Switch independent team Switch dependent team

Switch Independent Switch Dependent Address Hash Sends on all active members, receives on one member (primary member) Hyper-V port Sends on all active members, receives on all active members, traffic from same port always on same NIC Sends on all active members, All outbound traffic from a receives on all active port will go on a single NIC. members, inbound traffic may Inbound traffic may be use different NIC than distributed differently outbound traffic for a given depending on what the stream (inbound traffic is switch does to distributed by the switch) traffic

Switch Independent Switch Dependent Address Hash Hyper-V port Sends on all active members, receives on oneusing the members, on all Sends on all active members selected receives level of address hashing (primary (defaults to member) 4 -tuple hash). active members, traffic from member same port always on same Because each IP address can only be associated with a single NIC MAC address for routing purposes, this mode receives inbound Sends active members, All outbound trafficon onall only one member (the primary member). traffic from a receives on all active port will go on a single NIC. Best usedinbound when: members, traffic may Inbound traffic may be a) Native mode teaming where switch diversity is a concern; use different NIC than distributed differently b) Active/Standby mode outbound traffic for workloads a given thatdepending on whatlight the c) Servers running are heavy outbound, stream (inbound traffic is switch does to distribute inbound workloads (e. g. , IIS) distributed by the switch) traffic

Switch Independent Switch Dependent Address Hash Hyper-V port Sends on all active members, Sends on all active Sends on on all active the hashed Hyper-V switch receives port. Eachon Hyperreceives one members, all V pormembers using t will be bandwidth to nottraffic more from (primary member) active limited members, than one team member’s bandwidth. same port always on same Because each VM (Hyper-V port) is. NIC associated with a single NIC, this on mode inbound traffic. All for outbound the VM on the samefrom NIC it a Sends all receives active members, traffic sends on so all NICs receive inbound traffic. This also allows receives on all active port will go on a single NIC. maximum use of VMQs for better performance over all. members, inbound traffic may Inbound traffic may be use. Best different NIC than under the Hyper-V distributed used for teaming switchdifferently when - number VMsa well-exceeds number of team members outbound trafficoffor given depending on what the restriction traffic of a VMisto one NIC’s bandwidth is acceptable stream -(inbound switch does to distributed by the switch) traffic

Switch Independent Switch Dependent Address Hash Hyper-V port Sends on all active members, Sends on all active Sends on active members usingmembers, the selectedreceives level of address receives onall one member on all hashingmember) (defaults to 4 -tuple hash). active members, traffic from (primary same port always on same Receives on all ports. Inbound traffic is distributed by the switch. NIC and outbound traffic. There is no association between inbound Sends on all active members, All outbound traffic from a Best used for: receives on all active port will go on a single NIC. - Native teaming for maximum performance and switch members, traffic diversity inbound is not required; or may Inbound traffic may be use different than distributed differently - teaming. NIC under the Hyper-V switch when an individual VM needs to traffic be ablefor to transmit in excess ofon what one the team outbound a givenat rates depending what member can deliver stream (inbound traffic is switch does to distributed by the switch) traffic

Switch Independent Switch Dependent Address Hash Hyper-V port Sends on all active members, Sends on all active members using the hashed Hyper-V switch receives on one members, receives on all port. Each Hyper-V port will be bandwidth limited to not more (primary member) active members, traffic from than one team member’s bandwidth. same port always on same Receives on all ports. Inbound traffic NICis distributed by the switch. There is no association between inbound and outbound traffic. Sends on all active members, All outbound traffic from a receives on all active port will go on a single NIC. Best used when: - Hyper-V teaming whenmay VMs on the switch well-exceed the members, inbound traffic Inbound traffic may be of team members and distributed differently use number different NIC than - whentraffic policyfor callsafor e. g. , LACPdepending teams and when an the outbound given on what individual VM does not need to transmit faster than one team stream (inbound traffic is switch does to distribute member’s bandwidth distributed by the switch) traffic

Hyper-V switch VLAN =42 Default (all but 42) TEAM VLAN =42 VLAN =99 Default TEAM Black hole

Hyper-V switch Default TEAM

VLAN =13 VLAN =42 VLAN =99 TEAM VLAN =3995

Feature Comments RSS Programmed directly by TCP/UDP when bound to TCP/UDP. VMQ Programmed directly by the Hyper-V switch when bound to Hyper-V switch IPsec. TO, LSO, Jumbo frames, all checksum offloads (transmit) Yes – advertised if all NICs in the team support it RSC, all checksum offloads (receive) Yes – advertised if any NICs in the team support it DCB Yes – works independently of NIC Teaming RDMA, TCP Chimney offload No support through teaming SR-IOV Teaming in the guest allows teaming of VFs Network virtualization Yes

demo NIC Teaming

SMB Multichannel

Multiple connections per SMB session Full Throughput Bandwidth aggregation with multiple NICs Multiple CPUs cores engaged when using Receive Side Scaling (RSS) Sample Configurations Single 10 Gb. E RSS-capable NIC SMB Client Multiple 1 Gb. E NICs Multiple 10 Gb. E in a NIC team Multiple RDMA NICs SMB Client NIC Teaming RSS SMB Multichannel implements end-toend failure detection Leverages NIC teaming if present, but does not require it Automatic Configuration SMB detects and uses multiple network paths NIC 10 Gb. E NIC 1 Gb. E Switch 10 Gb. E Switch 1 Gb. E NIC 10 Gb. E RSS NIC 1 Gb. E SMB Server NIC 1 Gb. E Switch 1 Gb. E NIC 10 Gb. E Switch 10 Gb. E NIC 10 Gb. E/IB Switch 10 Gb. E/IB NIC Teaming SMB Server

SMB Client CPU utilization per core RSS Switch 10 Gb. E NIC 10 Gb. E SMB Server CPU utilization per core NIC 10 Gb. E RSS SMB Client RSS Core 1 Core 2 Core 3 Core 4 SMB Server Core 1 Core 2 Core 3 Core 4

1 session, without Multichannel SMB Client 1 RSS SMB Client 2 1 session, with Multichannel SMB Client 1 RSS SMB Client 2 RSS NIC 10 Gb. E NIC 10 Gb. E Switch 10 Gb. E NIC 10 Gb. E RSS SMB Server 1 SMB Server 2 RSS SMB Server 1 RSS SMB Server 2

SMB Client Interface Scaling - Throughput MB/sec 1 x 10 Gb. E Preliminary results based on Windows Server “ 8” Developer Preview 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0 512 1024 4096 2 x 10 Gb. E 8192 16384 3 x 10 Gb. E 32768 4 x 10 Gb. E 65536 131072 262144 524288 1048576 I/O Size http: //go. microsoft. com/fwlink/p/? Link. Id=227841

1 session, with NIC Teaming, no MC SMB Client 1 NIC Teaming NIC 10 Gb. E Switch 10 Gb. E NIC Teaming SMB Server 2 1 session, with NIC Teaming and MC SMB Client 2 NIC Teaming NIC 1 Gb. E SMB Client 1 NIC Teaming NIC 10 Gb. E Switch 1 Gb. E Switch 10 Gb. E Switch 1 Gb. E NIC Teaming SMB Server 2 Switch 10 Gb. E NIC Teaming SMB Server 1 SMB Client 2 NIC Teaming NIC 1 Gb. E Switch 1 Gb. E NIC Teaming SMB Server 2

1 session, without Multichannel SMB Client 1 SMB Client 2 1 session, with Multichannel SMB Client 1 SMB Client 2 R-NIC 54 Gb. IB R-NIC 10 Gb. E Switch 54 Gb. IB Switch 10 Gb. E R-NIC 54 Gb. IB R-NIC 10 Gb. E SMB Server 1 SMB Server 2

SMB Client NIC 1 Gb. E Wireless Switch 1 Gb. E NIC 1 Gb. E Wireless Single NIC configurations where full bandwidth is already available without MC Configurations with different NIC type or speed Wireless SMB Client RSS SMB Client SMB Server SMB Client NIC 10 Gb. E NIC 1 Gb. E R-NIC 10 Gb. E R-NIC 32 Gb. IB R-NIC 10 Gb. E NIC 1 Gb. E Wireless Switch 10 Gb. E Switch IB Switch 10 Gb. E Switch 1 Gb. E Wireless NIC 10 Gb. E NIC 1 Gb. E R-NIC 10 Gb. E R-NIC 32 Gb. IB R-NIC 10 Gb. E NIC 1 Gb. E RSS SMB Server NIC Switch SMB Server

Throughput Single NIC (no RSS) Fault Tolerance for SMB for non-SMB ▲ Multiple NICs (no RSS) ▲▲ ▲ Multiple NICs (no RSS) + NIC Teaming ▲▲ ▲▲ ▲ Single NIC (with RSS) Multiple NICs (with RSS) + NIC Teaming Single NIC (with RDMA) Multiple NICs (with RDMA) Lower CPU utilization ▲ ▲▲ ▲▲ ▲ ▲

demo SMB Multichannel

Virtualizing Storage for Scale, Resiliency, and Efficiency http: //go. microsoft. com/fwlink/? Link. ID=254536 How to Configure Clustered Storage Spaces in Windows Server 2012 http: //go. microsoft. com/fwlink/? Link. ID=254538 Storage Spaces FAQ http: //go. microsoft. com/fwlink/? Link. ID=254539