Intel Connects Cables Technology Improving Cluster Interconnect Reliability
Intel® Connects Cables Technology Improving Cluster Interconnect Reliability Otto Liebat April 14, 2008
Legal Disclaimers § Copyright © 2008 Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. § Other names and brands may be claimed as the property of their respective owners. Revision - 01 2
The Cluster Interconnect Problem 24 AWG Copper Cables § Distance Limited @DDR: 8 -10 meters* § Heavy: 1. 2 Kg for a 10 meter cable § Bulky: Blocks airflow, affects cooling Copper cables limit HPC cluster design Revision - 01 *Depends on BER requirement 3
Intel® Connects Cables What Are They ? Electrical Socket Optical Transceiver in Plug Optical Cable Up to 100 m Optical Transceiver in Plug A Drop In Replacement for Copper Cables § Compatible: Use existing copper sockets § Reliable: No user accessible optical interface § Low Cost: No separate optical transceivers Revision 01 testing Intel-internal *Source: 4
Intel® Connects Cables High-performance 20 Gbps Optical Cables Longer, Faster, Lighter, Thinner, … and More Reliable Revision - 01 5
Intel® Connects Cables Technology Highly Reliable: Real World Case Studies Ohio State University “One can now put switches and racks where they are needed, rather than within the typical 8 or 10 -meter reach of traditional cables. This will let us significantly expand the size of Infini. Band clusters, and that is one of our primary goals” -Dr. DK Panda Head of the OSU Network-Based Computing Research Group SCinet ’ 07 “The Intel cables were a clear success in our deployment. Every cable worked as intended, and that helped both the Open. Fabrics effort and SCinet deliver on our promises. ” Doug Fuller Co-Team Lead SCinet Open. Fabrics Subcommittee, SC ‘ 07 Revision - 01 6
Simple Cable Latency for Intel® Connects Cables • Optical/Electrical Conversions = 0. 275 nanoseconds each end • Speed of light through the fiber = 4. 99 nanoseconds per meter • Latency of a 10 meter Intel Connects Cable (ns) – First O/E conversion: – Speed of of light 10 m*4. 99 ns/m: – Second O/E conversion: – Total: 0. 275 49. 9 0. 275 50. 45 ns • Latency of a 100 meter Intel Connects Cable (ns) – First O/E conversion: – Speed of of light 100 m*4. 99 ns: – Second O/E conversion: – Total: 0. 275 499. 0 0. 275 499. 45 ns Note: 1 ns = 1 x 10 -9 sec Revision - 01 7
Revision - 01 Source: DK Panda, OSU
Effective Latency Function of –Simple Cable Latency –Bit Error Rate –The time required to find and fix those bit errors >Many things affect this • Other physical delays (e. g. passing through switches) • Where the error is detected • Is it the Bit Error random, or is there a bad link ? • Whether the data to be resent is in a buffer or has to be re-accessed from slower media. • The system and application tolerance for bit errors. Revision - 01 9
Intel® Connects Cables Better Quality: At 10 m. . at 100 m 1 Meter Intel® Connects Cable 10 Meter Intel® Connects Cable 1 m 100 Meter Intel® Connects Cable 5 Meter 24 AWG Copper Cable 100 m 5 m Superior signal quality from 1 to 100 meters Revision - 01 1 Source: Tektronix Lab Evaluation 10
Intel® Connects Cables Actual Bit Error Rates May Even Be Lower* 10 Meter Intel® Connects Cable** 10 -25 Extremely low BER for high HPC compute fabric stability Revision - 01 *Note: Specified BER for Intel® Connects Cables is 10 -15 **Source: Tektronix Lab Evaluation 11
Intel® Connects Cables More Reliable: 10 -15 BER Bit Error Rate at 20 Gbps per link 10 -12 BER @20 Gbps 10 -15 BER @20 Gbps Errors Per Day for a single link 1728 1. 7 Errors Per Day For 1000 links 1, 728, 000 10 -12 interconnects BER/day for 1000 links 1, 728 Intel® Connects Cable BER/day for 1000 links 1000 times less BER than interconnects at 10 -12 Revision - 01 *Source: Intel 12
Intel® Connects Cables Technology Highly Reliable: Real World Case Studies Computational Research Labs “We had to overcome significant reliability issues, but virtually all our reliability problems went away when we went with the Intel optical cables. ” -Ashrut Ambastha EKA Team Member, Computational Research Laboratories Navy Research Laboratory “Almost all our reliability problems went away when we went with the Intel optical cables. ” -Henry D. Dardy, Ph. D. Chief Scientist, Center for Computational Science, Naval Research Laboratory Revision - 01 13
Intel® Connects Cables High-Performance 20 Gbps Optical Cables Enable Large Cluster Scale Out § High data rate: 20 Gbps per cable* § Infini. Band or 10 Gb. E: CX 4 Connector § Long distance: Up to 100 meters § Low bit error rate: 10 -15 BER § Low conversion latency: 550 picoseconds** Reduced Installation, Maintenance § Less weight: 84% lighter § Less volume, better airflow: 83% smaller § Smaller bend radius: 40% less § Low Electro Magnetic Interference § No ground loops All claims Revision - 01 based on Intel Internal testing *Source: **Per pair of connectors 14
Intel® Connects Cables Technology 40 Gbps Ready Shown with Mellanox Connect. X* 40 Gbps Infini. Band* HCA Revision - 01 *Other names and brands may be claimed as the property of their respective owners. 15
Intel® Connects Cables Longer, Faster, Lighter, Thinner, … and More Reliable For more information visit… www. intelconnects. com* Revision - 01 16
- Slides: 16
