HighPerformance Networking With NDIS 6 0 TCP Chimney

High-Performance Networking With NDIS 6. 0, TCP Chimney Offload, and RSS Vik Desai Program Manager Windows Networking Microsoft Corporation

Appropriate Audience Who should attend this session? Networking product builders Product decision makers Hardware and software engineers Architects Network designers and deployers IT Managers IT Consultants Venture Capitalists and Private Investors Industry analysts

Agenda Networking stack challenges Scalable networking goals Scalable networking architecture Receive Side Scaling (RSS) TCP Chimney Offload Scalable networking demo Net. Xen Demo – Vikram Karvat Broadcom Demo – Uri Elzur Offload roadmap Summary and Call to Action

Networking Challenges Receive processing limited to a single CPU on a multi-processor system CPU utilized in Protocol Processing increases with Physical layer speeds Data Movement between network and application buffers is a bottleneck Large number of Interrupts even with Interrupt Moderation

Scalable Networking Goals Boost application scalability on 1 GB and 10 GB Ethernet with an integrated architecture That preserves standard infrastructure (1500 b MTU) That maintains standard network and server management practices That does not compromise security, server reliability, and application compatibility Enable Ethernet fabric convergence Robustly support new class of protocol offload NICs in Microsoft Windows

Receive Side Scaling Networking Challenge Receive processing limited to a single CPU on a multi-processor system Solution Parallelize receive processing by Queuing incoming packets to multiple CPUs Implementing Solution via RSS NIC manages multiple hardware queues NIC hashes incoming TCP segments to different hardware queues NIC driver requests DPCs on appropriate CPUs

RSS Description – Non RSS Capable NIC Processor 0 APP D P C ISR Interrupt Logic TCPIP NDIS Receive FIFO Incoming Packet Regular NIC

RSS Description – RSS Capable NIC Processor 0 Processor 1 APP D P C ISR Processor 2 APP D P C TCPIP NDIS RSS Capable. NIC Interrupt Logic Receive FIFOs Toeplitz Hash Incoming Packet

TCP Chimney Offload Networking Challenges Data Movement between network and application buffers is a bottleneck Large number of Interrupts even with Interrupt Moderation CPU utilized in Protocol Processing increases with Physical layer speeds Solution Provide Zero Copy solution for pre posted buffers Change interrupts from a per packet basis to a per segment basis Offload Protocol Processing to hardware

TCP Chimney Architecture Applications Other Misc. Layers Switch TCP Chimney Interfaces Path Layer IPv 4 or IPv 6 State Updates Framing Layer (Ethernet) NDIS 5. 2 / 6. 0 NDIS Miniport Driver TCP Chimney Offload Capable Hardware Data Transfer Transport Layer (TCP)

TCP Chimney Interface Details TCP/IP States Divided into Const State – Does not change for connection lifetime Cached State – Controlled by host stack and updated appropriately to offload target Delegated State – Controlled by Offload Target NDIS Supports Offload Capability Advertisement Interface to transfer and update state information Interface to query statistics Interface to transfer data

TCP Chimney Initialization Offload Manager determines suitability of connection for offload State from each layer is captured and transferred to offload target Incoming Data packets/outgoing sends are queued Data packets will be replayed to offload targets for successful offload attempts Data packets will be processed by stack for unsuccessful offload attempts Data Transfer Begins

TCP Chimney Data Transfer Sends Segment passed to offload target for completion Send Completions after end-to-end TCP Ack Receive If no receive buffers posted indicate data If receive buffers are posted indication occurs as appropriate OOB/Urgent Data passed to Host Stack

TCP Chimney Connection Teardown Connections can be uploaded/offloaded at any time Heuristics Manager tracks connections appropriate for upload/offload Half Closed Connections are not uploaded Upload request initiated by offload target Offload target to provide delegated state to host stack Offload target keeps connection state till host sends upload call

TCP Chimney Implications IPsec Chimney required for IPsec traffic Will not work with IM drivers incapable of understanding Chimney interfaces Hooking Firewalls Best benefits for Long Lived Connections Pre-posted Receive Buffers Large Application IO Sizes

10 Gb. E Chimney Offload Vikram Karvat VP Marketing vkarvat@netxen. com Faisal Latif Principal Software Engineer faisal@netxen. com

Net. Xen Next generation Ethernet silicon provider focused on server OEMs Chips, Boards, S/W Founded February 2002 Top tier investors Accel, Benchmark, Integral Capital Expertise in semiconductor, software, systems and servers Intelligent NIC™ product line Launched March 27, 2006 REAL products, REAL customers

Intelligent NIC Architecture DDR Gb. E CAM L 2 Caches QM CORE INTERCONNECT FABRIC Flow Classifier 10 GE 10 Gb. E QDR PCI-E 8 X Protocol Processing Engine Single-Chip Dual 10 Gb. E Quad Gb. E Protocol Features TCP/IP RDMA i. SCSI Virtualization Security Native 8 X PCI-express 1 X/4 X/8 X

Net. Xen 10 Gb. E Chimney Rx 3. 4 GHz Xeon Tx 10 Gb. E Switch Windows Server 2003 SP 1 with SNP

10 Gb. E Chimney Results 60% Throughput 800% Processor Efficiency CPU Utilization Configuration: DP Xeon, 3. 4 GHz, HT off, 2 GB

Demo Conclusion 10 Gb. E is happening NOW Chimney enables Scalability with balanced system design Increased datacenter power efficiency The Agile Datacenter requires Adaptability, Scalability, Intelligence

Broadcom Uri Elzur Director, Advanced Technology Broadcom Gururaj Ananthateerta Senior Staff Engineer Broadcom

Scalable TCP Chimney enables Convergence Over Ethernet Scalable TCP Chimney - basis for Convergence over Ethernet TCP based - Socket applications, i. SCSI boot, i. WARP (RDMA) Microsoft’s SNP enable convergence over Ethernet Sockets Applications Secure (Network based security), robust Windows Sockets and standard compliant Windows Socket Switch implementation is required Storage RDMA Provider Applications Ethernet requires Layer 2 functionality – VLAN, Wo. L, power management Integrated Management Kernel Mode File System Partition TCP/IP Class Driver NDIS i. SCSI Port Driver (iscsiprt. sys) NDIS IM Driver i. SCSI Miniport NDIS Miniport HBA NIC C-NIC User Mode RDMA Driver RNIC

Broadcom’s C-NIC 2. 5 G/S NTTCP over 2. 5 GB/s TCP Chimney NTTTCPs C-NIC Perfmon BCM 5708 S fiber cable S 2 (TX/RX) HP DL 380 G 4 server 3. 4 GHz Intel Xeon CPU 1 GB RAM Windows Server 2003 SP 1 -SNP build 2670 Two BCM 5708 S NICs Broadcom Miniport driver v 2. 6. 14* S 1 (TX/RX) Broadcom 2. 5 G Switch BCM 56580 Strata. XGS III HP DL 380 G 4 server 3. 4 GHz Intel Xeon CPU 1 GB RAM Windows Server 2003 SP 1 -SNP build 2670 Two BCM 5708 S NICs Broadcom Miniport driver v 2. 6. 14*

TCP Chimney scales… Higher is better TOE L 2 BW improvement TOE vs. L 2 Lower is better TOE L 2 CPU Utilization reduction TOE vs. L 2 • 2. 5 G/S offers more BW than non-TOE, at 1/6 of the CPU utilization • Microsoft’s SNP combined with BCM 5708 provides 7. 5 times better P/E • Performance Efficiency (P/E) is network throughput divided by CPU Utilization • At Gigabit and beyond, TCP Chimney is critical to free up cycles for the applications Demo: NTTTCP

RSS Improves SMP Scalability Demo: Web Bench 5. 0 With RSS web traffic is more evenly distributed on multiple CPUs Web Bench delivers up to 50% more requests/sec

Demo Conclusion Broadcom’s C-NIC with Microsoft’s TCP Chimney is here TODAY TCP Chimney scales to accommodate the needs of the server and applications TCP Chimney is the basis for the future of Networking in Windows Architecture allows for IPsec based security RSS provides for a better load spreading on SMP servers

Scalable Networking Pack Partners

Future Chimney Offloads IPsec Chimney RDMA Chimney SSL Chimney

Call To Action Develop low cost TCP Chimney Offload and RSS hardware for Windows Vista and Windows Server codenamed “Longhorn” Deploy TCP Chimney Offload and RSS hardware in enterprise and personal computing environments

Additional Resources Web Resources Documentation, White Papers, and software bits available today for TCP Chimney Offload and RSS: http: //support. microsoft. com/? kbid=912222 Specs: DDK and Documentation will available on: www. microsoft. com/whdc White Paper: http: //www. microsoft. com/whdc/device/network/scale. mspx Other Resources: www. microsoft. com/snp http: //www. microsoft. com/whdc/device/network/netintro. mspx Related Sessions Net 088 – Technical Overview of Microsoft’s Net. DMA Architecture Please send e-mail tondis 6 fb @ microsoft. com with questions

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.