Initial Performance Measurements Gigabit Ethernet NICs 64 bit
Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress) Collaboration: Boston Ltd. (Watford) – Super. Micro Motherboards, CPUs, Intel GE NICs Brunel University – Peter Van Santen University of Manchester – Richard Hughes-Jones GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
The Measurements (1) u Latency u Round trip times measured using Request-Response UDP frames u Latency as a function of frame size n Slope gives sum of individual data transfer rates end-to-end n Mem copy + pci + Gig Ethernet + pci + mem copy u Histograms of individual measurements GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
The Measurements (2) u UDP Throughput u Send a burst of UDP frames spaced at regular intervals u Vary the frame size and the frame transmit spacing u Record n n The time to send and the time to receive the frames The number received, the number lost, number out of order The received inter-packet spacing CPU load, Number of interrupts Zero stats Send data frames at regular intervals OK done ●●● Inter-packet time Time to receive Time to send Get remote statistics Send statistics GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
The Measurements (3) u PCI Activity u Logic Analyzer with Gigabit Ethernet Probe chipset NIC CPU NIC n PCI Probe cards in sending PC n Gigabit Ethernet Fiber Probe Card n PCI Probe cards in receiving PC CPU chipset mem Logic Analyser Display GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Latency: Alteon Ace. NIC n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
UDP Throughput: Alteon Ace. NIC n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Alteon Ace. NIC n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit n Red. Hat 7. 1 Kernel 2. 4. 14 n n ALT 33102 PCI 33 MHz 1400 bytes sent Wait 16 us Send PCI Receive PCI n n n ALT 66101 66 MHz 1400 bytes sent Wait 16 us NIC cannot sustain 66 MHz GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Latency: Sys. Konnect n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 33 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n Latency low good Latency well behaved Slope 0. 0252 us/byte Expect: PCI 0. 00758 Gig. E 0. 008 PCI 0. 00758 0. 0236 us/byte GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
UDP Throughput: Sys. Konnect n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 33 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n Max throughput 690 Mbit/s n No packet loss n Packet loss during drop GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Sys. Konnect n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n SK 300 1400 bytes sent Wait 100 us ~8 us for send or receive Gigabit Ethernet frame GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Sys. Konnect n Motherboard: Super. Micro 370 DLE Chipset: Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n SK 301 n 1400 bytes sent n Wait 20 us n n n Sk 303 1400 bytes sent Wait 10 us Frames are back-to-back Cannot go any faster ! Gig Eth frames back to back GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Latency: Intel n Motherboard: Super. Micro 370 DLE Chipset: : Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n Latency high Latency well behaved Slope 0. 0187 us/byte Expect: PCI 0. 00188 Gig. E 0. 008 PCI 0. 00188 0. 0118 us/byte GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Intel n Motherboard: Super. Micro 370 DLE Chipset: : Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n n IT 66 M 200 64 bytes sent CSR time: 1. 75 us Data time 0. 25 us Interrupt delay: ~70 us n 1400 response GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Throughput: Intel n Motherboard: Super. Micro 370 DLE Chipset: : Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n Max throughput 910 Mbit/s n No packet loss n Packet loss during drop GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Throughput: Intel n Motherboard: Super. Micro 370 DLE Chipset: : Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n losses occur in groups n ~50 pkts every 140 GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Intel n Motherboard: Super. Micro 370 DLE Chipset: : Server. Works III LE Chipset n CPU: PIII 800 MHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n n IT 66 M 212 1400 bytes sent Wait 11 us ~4. 7 us on send PCI bus ~45% occupancy ~ 3. 25 us on PCI for data recv n n n IT 66 M 212 1400 bytes sent Wait 11 us Packets lost Action of pause packet? GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Latency: Intel on P 4 CD 6+ n Motherboard: Super. Micro P 4 CD 6+ Chipset: Intel i 860 n CPU: Dual Xeon Prestonia (2 cpu/die) 2. 2 GHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 2 n Latency high n Slope 0. 0195 us/byte n Expect: PCI 0. 00188 Gig. E 0. 008 PCI 0. 00188 0. 0118 us/byte GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Throughput: Intel on P 4 CD 6+ n Motherboard: Super. Micro P 4 CD 6+ Chipset: Intel i 860 n CPU: Dual Xeon Prestonia (2 cpu/die) 2. 2 GHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 2 n Max throughput 950 Mbit/s n No packet loss n Negligible Packet loss GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Intel on P 4 CD 6+ n Motherboard: Super. Micro P 4 CD 6+ Chipset: Intel i 860 n CPU: Dual Xeon Prestonia (2 cpu/die) 2. 2 GHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 2 n n n IT 66 M 220 1400 bytes sent Wait 1000 us CSR time: 12. 25 us Data time 0. 25 us Interrupt delay: ~79 us n n n IT 66 M 224 1400 bytes sent Wait 100 us Detail Chipset limits PCI transfers with STOPs n Try i 870 Chipset GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Intel on P 4 CD 6+ n Motherboard: Super. Micro P 4 CD 6+ Chipset: Intel i 860 n CPU: Dual Xeon Prestonia (2 cpu/die) 2. 2 GHz PCI: 64 bit 66 MHz n Red. Hat 7. 1 Kernel 2. 4. 2 n IT 66 M 221 n 1400 bytes sent n Wait 11 us GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Latency: Intel on IBM board n Motherboard: IBM das Chipset: : Server. Works CNB 20 LE n CPU: Dual PIII 1 GHz PCI: 64 bit 33 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n Latency high Latency well behaved Slope 0. 0206 us/byte Expect: PCI 0. 00376 Gig. E 0. 008 PCI 0. 00376 0. 0155 us/byte GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
Throughput: Intel on IBM board n Motherboard: IBM das Chipset: : Server. Works CNB 20 LE n CPU: Dual PIII 1 GHz PCI: 64 bit 33 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n Max throughput 930 Mbit/s n No packet loss n Packet loss during drop GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
PCI: Intel on IBM board n Motherboard: IBM das Chipset: : Server. Works CNB 20 LE n CPU: Dual PIII 1 GHz PCI: 64 bit 33 MHz n Red. Hat 7. 1 Kernel 2. 4. 14 n n n uva 64 m 02 1400 bytes sent Wait 11 us ~9. 3 us on send PCI bus ~82% occupancy ~ 5. 9 us on PCI for data recv. GGF 4 Toronto Feb 2002 R. Hughes-Jones Manchester
- Slides: 23