CCS Network Roadmap Presented by Steven M Carter

CCS Network Roadmap Presented by Steven M. Carter Network Task Lead Center for Computational Sciences

CCS network roadmap Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement · FY 2006–FY 2007 - Scaled Ethernet to meet wide-area needs and current day local-area data movement - Developed wire-speed, low-latency perimeter security to fully utilize 10 G production and research WAN connections - Began IB LAN deployment · FY 2007–FY 2008 - Continue building out IB LAN infrastructure to satisfy the file system needs of the Baker system - Test Lustre on IB/WAN for possible deployment 2

NCCS network roadmap summary Ethernet core scaled to match wide-area connectivity and archive Lustre Infiniband core scaled to match central file system and data transfer Baker Ethernet [O(10 GB/s)] Gateway Jaguar High-Performance Storage System (HPSS) Viz 3 Infiniband [O(100 GB/s)]

CCS WAN overview Ciena Core. Stream Ciena Core. Director Tera. Grid Juniper T 320 DOE Ultra. Science/ NSF CHEETAH Force 10 E 600 Juniper T 640 CCS Cisco 6509 NSTG CCS OC-192 to Tera. Grid OC-192 to Internet 2 2 x OC-192 to USN OC-192 to Esnet OC-192 to CHEETAH Internal 10 G Link Internal 1 G Link 4

CCS Network 2007 Infiniband Ethernet 48 -96 SDR Jaguar 128 DDR 32 Spider 60 48 48 48 Viz 48 48 48 64 DDR 32 DDR HPSS 20 4 48 4 64 DDR 24 SDR Devel 87 SDR 48 E 2 E 48 3 SDR 20 SDR Spider 10 5 20

CCS IB network 2008 20 SDR 24 SDR Spider 10 Devel 87 SDR Jaguar E 2 E 48 -96 SDR (16 -32 SDR/link) 64 DDR Viz 50 DDR 32 DDR HPSS 50 DDR 300 DDR (50 DDR/link) Baker 6 Spider 240

FY 2007 milestones Nov 2006 - Completed IB WAN testing Jan 2007 - Completed 10 -G WAN upgrade Jan 2007 - IBo. IP firewall installation Mar 2007 - Force 10 P 10 upgrade Apr 2007 - Completed VL/QOS testing Jun 2007 - IBo. IP/SDP decision point 7

IB over WAN testing · Placed 2 x Obsidiam Longbow devices between Voltaire 9024 and Voltaire 9288 Ciena CD-CI (SNV) DOE Ultra. Science Network Obsidian Longbow Ciena CD-CI (ORNL) Voltaire 9024 20 · RDMA Test Results: Local: 7. 5 Gbps (Longbow to Longbow) ORNL <-> ORNL (0. 2 mile): 7. 5 Gbps ORNL <-> Chicago (1400 miles): 7. 46 Gbps ORNL <-> Seattle (6600 miles): 7. 23 Gbps ORNL <-> Sunnyvale (8600 miles): 7. 2 Gbps Voltaire 9288 4 x Infiniband SDR 81 OC-192 SONET Lustre 8 · Provisioned loopback circuits of various lengths on the DOE Ultra. Science Network and ran test. End-to-End

Network Task Lead Voltaire 9024 Rizzo (XT 3) Spider (Linux Cluster) The basic RDMA test (as provided by the Open Fabrics Enterprise Distribution), allows us to see the maximum bandwidth that we could achieve (about 900 MB/s unidirectionally) from the XT 3's 133 MHz PCI-X bus. 9 Utilizing a server on spider (a commodity x 86_64 Linux cluster), we were able to show the first successful infiniband implementation on the XT 3.

Contact Steven Carter Tech Integration Center for Computational Sciences (865) 576 -2672 scarter@ornl. gov 10 Presenter_date