Longbow Infini Band Extension Dr David T Southwell

Longbow Infini. Band Extension Dr. David T Southwell President & CEO Obsidian Research Corporation - CERN June 26 th, 2006

Agenda • The fundamentals of Infini. Band flow control • Infini. Band range limitations – two mechanisms • Longbow Infini. Band range extension technology • Potential applications at CERN Obsidian Research Corporation - CERN June 26 th, 2006 2

Infini. Band flow control • Infini. Band is credit based • On initialisation, each fabric end-point declares its capacity to receive data • This capacity is described as it’s “buffer credit” • As buffers are freed up, end points post messages updating their credit status • Infini. Band is therefore lossless – data is never thrown away, since… …Infini. Band flow control happens before the transmission, not after it! • Note the buffer credit mechanism applies to every point-to-point link (not end-to-end) • This mechanism is in contrast to Ethernet’s loss-based “flow control” : • On network over-subscription, packets are simply thrown away • Detected packet loss triggers retransmissions and adjustments to the injection rate Obsidian Research Corporation - CERN June 26 th, 2006 3

Infini. Band range limitations • As commercialised today, Infini. Band addresses the cluster/ supercomputer market… NASA’s “Columbia”– 10, 240 Itaniums (NUMALink+Infini. Band interconnect) • High equipment packing density (rack-to-rack connections are short) • Infini. Band switches cascade easily (very low latency), so multi-hop is ok • High port count switches (large ICs) Obsidian Research Corporation - CERN June 26 th, 2006 4

Mechanism (1) – physical layer • These applications are served by standard Infini. Band cables: • Balanced copper cables (twin-axial, shielded & tight impedance control) • Cheaper than optics, but with a range < 20 m (RF losses) @ 2. 5 GBits/s (“SDR”) • At DDR (5 Gbits/s) and QDR (10 Gbits/s) rates per channel, cables get even shorter (today) (soon!) NASA’s Columbia – 10, 240 Itaniums • There exists a parallel optic multi-mode fibre solution (simple E-O-E) (NUMALink+Infini. Band interconnect) • More expensive (especially the parallel fibre bundles themselves) • Self-limits @ ~200 m • A good solution for longer inter-rack runs or for links between floors • MPO will see more use at DDR/ QDR rates Obsidian Research Corporation - CERN June 26 th, 2006 5

Mechanism (2) – link layer • Optimised for a short signal flight time; small buffers are used inside the ICs: • Facilitates switch IC implementation, but limits effective range to ~ 300 m Undersized buffers restrict the sustained data flow rate – in this case data is only moving in phases 1 and 5! The inefficiency is caused by an inability to keep the pipe full by restoring the receive credits fast enough to avoid a break up of the burst. NASA’s Columbia – 10, 240 Itaniums The longer the flight (NUMALink+Infini. Band interconnect) transfer rate is. time, the lower the effective This limits the useful length of an Infini. Band link no matter what the physical transport is capable of. (Nb. this has no impact on copper Infini. Band links – receive buffers >> 2 x wire data capacity). Obsidian Research Corporation - CERN June 26 th, 2006 6

Longbow Technology Obsidian has developed a technology that performs Infini. Band encapsulation over 10 Gb. E, Packet Over SONET/SDH and ATM WANs at 4 x Infini. Band speeds: Longbow XR. • Looks like a 2 -port Infini. Band switch to the Infini. Band fabric • Designed for 100, 000 km+ ranges, prototypes publicly tested over 1, 500 km and 8, 500 km OC-192 c networks (SC|04, OFC’ 05, SC|05) 950+ MBytes/s sustained performance in a single logical flow ~ 4% CPU load (Opteron 242 s using RDMA transport) IPv 6 Packet Over SONET & ATM modes NASA’s Columbia – 10, 240 Itaniums (NUMALink+Infini. Band interconnect) Obsidian Research Corporation - CERN June 26 th, 2006 7

Longbow Transport NASA’s Columbia – 10, 240 Itaniums (NUMALink+Infini. Band interconnect) Obsidian Research Corporation - CERN June 26 th, 2006 8

Longbow @ SC|05 NASA’s Columbia – 10, 240 Itaniums (NUMALink+Infini. Band interconnect) Obsidian Research Corporation - CERN June 26 th, 2006 9

The Obsidian Longbow XR • Transparent to Infini. Band hardware, stacks and applications NASA’s Columbia – 10, 240 Itaniums • Very (NUMALink+Infini. Band user-friendly long-haulinterconnect) wire-speed Infini. Band data pump • Compatible with all Infini. Band equipment and stacks, including Open. Fabrics • High availability architecture – telecom grade equipment • A managed device (HTTP GUI, SSH CLI, SNMP) – 10/100 Ethernet/ serial console • Also encapsulates two Gb. Ethernet channels along with the 4 x SDR Infini. Band channel Obsidian Research Corporation - CERN June 26 th, 2006 10

Potential Application…ATLAS In collaboration with Dr. Bryan Caron (University of Alberta, Canada), Bill St. Arnaud (Canarie Inc. - Canada’s high performance research network) and others, Obsidian will soon launch a multi-stage Long Haul Infini. Band project which will demonstrate reliable, 10 Gbits/s transfer of bulk data back & forth across the Atlantic: CERN would be the preferred end point for such a demonstration - Canarie has confirmed that the entire lightpath would be available for sustained streaming demonstrations. Obsidian Research Corporation - CERN June 26 th, 2006 11

Longbow Campus and Metro Obsidian also sees application for the range extension technology over SONET/ SDH networks for Metro Area Networks (up to 120 km), and for dark fibre campus applications (up to 10 km). • Remote Infini. Band storage (replication, distributed SAN) • Visualisation applications; tap directly and natively into distant clusters • Aggregate remote Infini. Band clusters into larger compute resources Campus and Metro versions are currently in development. They will be optimised for latency and the more efficient use of smaller networks. Obsidian Research Corporation - CERN June 26 th, 2006 12

Conclusions Infini. Band is becoming a critical element in high performance computing architectures. With demonstrated uncompromising long haul capability, Infini. Band Longbow technology may represent an excellent long term platform for globally distributing the relentless data streams LHC will emit during its lifetime. Infini. Band, global optical network transports and Longbow technologies will scale in performance over time to continue to offer a compelling system-level solution that will present a stable interface to the applications software. Thank you for your attention. http: //www. obsidianresearch. com (P. S. Thanks for Web too!) Obsidian Research Corporation - CERN June 26 th, 2006 13