Checksum offloading A look at how the Pro

Checksum ‘offloading’ A look at how the Pro 1000 NICs can be programmed to compute and insert TCP/IP checksums

Network efficiency • Last time (in our ‘nictcp. c’ demo) we saw the amount of work a CPU would need to do when setting up an ethernet packet for transmission with TCP/IP protocol format • In a busy network this amount of packetcomputation becomes a ‘bottleneck’ that degrades overall system performance • But a lot of that work can be ‘offloaded’!

The ‘loops’ are costly • To prepare for a packet-transmission, the device-driver has to execute a few dozen assignment-statements, to set up fields in the packet’s ‘headers’ and in the Transmit Descriptor that will be used by the NIC • Most of these assignments involve simple memory-to-memory copying of parameters • But the ‘checksum’ fields require ‘loops’

Can’t ‘unroll’ checksum-loops • One programming technique for speeding up loop-execution is known as ‘unrolling’, to avoid the ‘test-and-branch’ inefficiency: int sum = 0; sum += wp[0]; sum += wp[1]; sum += wp[2]; … sum += wp[99]; • But it requires knowing in advance what number of loop-iterations will be needed

The ‘offload’ solution • Modern network controllers can be built to perform TCP/IP checksum calculations on packet-data as it is being fetched from ram • This relieves a CPU from having to do the most intense portion of packet preparation • But ‘checksum offloading’ is an optional capability that has to be ‘enabled’ – and ‘programmed’ for a specific packet-layout

‘Context’ descriptors • Intel’s Pro 1000 network controllers employ special ‘Context’ Transmit-Descriptors for enabling and configuring the ‘checksumoffloading’ capability • Two kinds of Context Descriptor are used: – An ‘Offload’ Context Descriptor (Type 0) – A ‘Data’ Context Descriptor (Type 1)

Context descriptor (type 0) 63 48 47 40 39 TUCSE TUCSO MSS HDRLEN 32 31 TUCSS RSV 16 15 IPCSE STA TUCMD DTYP =0 8 IPCSO 7 0 IPCSS PAYLEN DEXT=1 (Extended Descriptor) Legend: IPCSS (IP Check. Sum Start) IPCSO (IP Check. Sum Offset) IPCSE (IP Check. Sum Ending) PAYLEN (Payload Length) TUCMD (TCP/UCP Command) HDRLEN (Header Length) TUCSS (TCP/UDP Check. Sum Start) TUCSO (TCP/UDP Check. Sum Offset) TUCSE (TCP/UDP Check. Sum Ending) DTYP (Descriptor Type) STA (TCP/UDP Status) MSS (Maximum Segment Size)

The TUCMD byte 7 IDE 6 5 4 SNAP DEXT (=1) reserved (=0) 3 RS 2 TSE Legend: IDE (Interrupt Delay Enable) SNAP (Sub-Network Access Protocol) DEXT (Descriptor Extension) RS (Report Status) TSE (TCP-Segmentation Enable) IP (Internet Protocol) TCP (Transport Control Protocol) always valid only when TSE=1 1 IP 0 TCP

Context descriptor (type 1) 63 48 47 40 39 32 31 16 15 8 7 ADDRESS VLAN POPTS RSV STA DCMD DTYP =1 DTALEN DEXT=1 (Extended Descriptor) Legend: DTALEN (Data Length) DTYP (Descriptor Type) DCMD (Descriptor Command) STA (Status) RSV (Reserved) POPTS (Packet Options) VLAN (VLAN tag) 0

The DCMD byte 7 IDE 6 5 4 VLE DEXT (=1) reserved (=0) 3 RS 2 TSE Legend: IDE (Interrupt Delay Enable) VLE (VLAN Enable) DEXT (Descriptor Extension) RS (Report Status) TSE (TCP-Segmentation Enable) IFCS (Insert Frame Check. Sum) EOP (End Of Packet)) always valid only when EOP=1 1 IFCS 0 EOP

Our usage example • We’ve created a module named ‘offload. c’ which demonstrates the NIC’s checksumoffloading capability for TCP/IP packets • It’s a modification of our earlier ‘nictcp. c’ character-mode device-driver module • We have excerpted the main changes in a class-handout – the full version is online

Data-type definitions // Our type-definition for the ‘Type 0’ Context-Descriptor typedef struct { unsigned char unsigned short ipcss; ipcso; ipcse; unsigned char unsigned short tucss; tucso; tucse; unsigned int paylen: 20; dtyp: 4; tucmd: 8; unsigned char status; unsigned char hdrlen; unsigned short mss; } TX_CONTEXT_OFFLOAD;

Definitions (continued) // Our type-definition for the ‘Type 1’ Context-Descriptor typedef struct { unsigned long base_addr; unsigned int dtalen: 20; dtyp: 4; dcmd: 8; unsigned char status; unsigned char pkt_opts; unsigned short vlan_tag; } TX_CONTEXT_DATA; typedef union { TX_CONTEXT_OFFLOAD TX_CONTEXT_DATA } TX_DESCRIPTOR; off; dat;

Our packets’ layout Ethernet Header (14 bytes) IP Header (20 bytes) TCP Header (20 bytes) Packet-Data (length varies) 14 bytes 10 bytes 16 bytes (no options) HDR CKSUM TCP CKSUM (no options)

How we use contexts • Our ‘offload. c’ driver will send a ‘Type 0’ Context Descriptor within ‘module_init()’ txring[ 0 ]. off. ipcss = 14; txring[ 0 ]. off. ipcso = 24; txring[ 0 ]. off. ipcse = 34; // IP-header Check. Sum Start // IP-header Check. Sum Offset // IP-header Check. Sum Ending txring[ 0 ]. off. tucss = 34; txring[ 0 ]. off. tucso = 50; txring[ 0 ]. off. tucse = 0; // TCP/UDP-segment Check. Sum Start // TCP/UDP-segment Checksum Offset // TCP/UDP-segment Checksum Ending txring[ 0 ]. dtyp = 0; // Type 0 Context Descriptor txring[ 0 ]. tucmd = (1<<5)|(1<<3); // DEXT=1, RS=1 iowrite 32( 1, io + E 1000_TDT ); // give ownership to NIC

Using contexts (continued) • Our ‘offload. c’ driver will then use a Type 1 context descriptor every time its ‘write()’ function is called to transmit user-data • The network controller ‘remembers’ the checksum-offloading parameters that we sent during module-initialization, and so it continues to apply them to every outgoing packet (we keep our same packet-layout)

Sequence of ‘write()’ steps • • Adjust the ‘len’ argument (if necessary) Copy ‘len’ bytes from the user’s ‘buf’ array Prepend the packet’s TCP Header Insert the pseudo-header’s checksum Prepend the packet’s IP Header Prepend the packet’s Ethernet Header Initialize the Data-Context Tx-Descriptor Give descriptor-ownership to the NIC

The TCP pseudo-header • We do initialize the TCP Checksum field, (but this only needs a short computation) Zero Protocol-ID TCP Segment-length (= 6) Source IP-address Destination IP-address • The one’s complement sum of these six words is placed into ‘TCP Checksum’

Setting up the Type-1 Context int txtail = ioread 32( io + E 1000_TDT ); txring[ txtail ]. dat. base_addr = tx_desc + (txtail * TX_BUFSIZ); txring[ txtail ]. dat. dtalen = 54 + len; txring[ txtail ]. dat. dtyp = 1; txring[ txtail ]. dat. dcmd = 0; txring[ txtail ]. dat. status = 0; txring[ txtail ]. dat. pkt_opts = 3; // IXSM=1, TXSM=1 txring[ txtail ]. dat. vlan_tag = vlan_id; txring[ txtail ]. dat. dcmd |= (1<<0); txring[ txtail ]. dat. dcmd |= (1<<3); txring[ txtail ]. dat. dcmd |= (1<<5); txring[ txtail ]. dat. dcmd |= (1<<6); txtail = (1 + txtail) % N_TX_DESC; iowrite 32( txtail, io + E 1000_TDT ); // EOP (End-Of-Packet) // RS (Report Status) // DEXT (Descriptor Extension) // VLE (VLAN Enable)

In-class demonstration • We can demonstrate checksum-offloading by using our ‘dram. c’ device-driver to look at the packet that is being transmitted from one of our ‘anchor’ machines, and to look at the packet that gets received by another ‘anchor’ machine • The checksum-fields (at offsets 24 and 50) do get modified by the network hardware!

In-class exercise • The NIC can also deal with packets having the UDP protocol-format – but you need to employ different parameters in the Type 0 Context Descriptor and arrange a ‘header’ for the UDP segment that has a different length and arrangement of parameters • Also the UDP protocol-ID is 17 (=0 x 11)

UDP Header 0 0 0 0 0 1 1 1 1 1 2 2 2 2 2 3 3 0 1 2 3 4 5 6 7 8 9 0 1 0 Source Port Destination Port Length Checksum Data : : : Traditional ‘Big-Endian’ representation