The DOE Ultra Science Network aka Ultra Net

The DOE Ultra. Science Network (aka Ultra. Net) GNEW 2004 March 15, 2004 Wing/Rao OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Talk Outline · · · What are we trying to do? Why do it? How (we all sort of know don’t we)? How can we get there from here? …Where will that be? OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

DOE Science · DOE has bigger problems (drivers) … (and less money) than any other threeletter agency · But with it, we are trying to do things like: - Climate Modeling - Supernova modeling - Genomics (and proteomics) OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Climate Modeling North American model at 400 Km resolution North American model at 60 Km resolution The Rockies block the atmospheric circulation and are important geneses regions for the weather systems that form the North Atlantic Storm Track. They are poorly represented in low resolution models. OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Type II Core. Collapse Supernova OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Computational Molecular Biology. 25 TF Membrane with possible uses for bioremediation 11 TF 400 TF >1. 0 PF Molecular dynamics of a lipopolysaccharide (LPS) Nanosecond molecular dynamics of the LPS membrane of Pseudomonas aeruginosa QM/MM molecular dynamics of membrane plus mineral OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Energy transduction across membranes is still out of reach at 1. 0 PF

Billions of Bases in Gen. Bank According to the GOLD database, there are 146 published genomes, 344 prokaryotic ongoing genomes projects, and 243 eukaryotic ongoing genome projects. 1982 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 1986 1990 1994 1998 2002

Growth of Proteomic Data vs. Sequence Data OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Peta. Scale Science · Climate modeling today => 50 -100 TB data sets - Still adding carbon chemistry - Still refining resolution · You saw DOE Road Map table showing PB requirements in 2 -3 years · Supernova modeling just going from 2 1/2 dimensions to 3 D - this is a PB problem today · Genomics and Proteomics will dwarf these · …and then of course there is High Energy …but OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

We run hard into network limits… · There is some VERY nice work being done on TCP - But that will just emphasize router delay - And expose congestion, even where it coexists with it · There is nice work being done to extend MPLS - But that does not eliminate jitter - And imposes additional overhead · And there is still the tyranny of layers OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Jitter - ORNL / Atlanta OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Science Applications - Bit Rate Budget End-to-end Bit-Rate 100 Gbps DWDM (One Lambda) Research Opportunities 40 Gbps OC-768 10 Gbps OC 192 ESnet 5. 0 Gbps 2. 5 Gbps SONET (One Channel) TCP/IP (One Flow) Site Security App s ESnet SITE LAN Host ESnet Optical Backbone IP Network Services Host OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY End-to-end Performance Gap Firewall OC-48 1. 0 Gbps. 65 Gbps Cluster Gig. E One Host

What are we trying to do? · We are trying to break out of the limitations of conventional networks - TCP/IP - Firewalls, and layers - Small-frame technologies · Take advantage of the real throughput available in optical networks today at the SONET layer · Develop the techniques needed within 2 -3 years for DOE Peta-scale science · Use switched-circuits as a solution we can start on today OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Restating the Problem · Networks will soon offer 40 Gbs channels · Typical multi-stream throughput limited to 5 -6 Gbs, and programming multi-steam is hard - hero efforts yield 9 -10 Gbs · Typical single-steam throughput to a single application ~1 Gbs, even on a tuned network - hero effort yields 5 -6 Gbs · Firewalls throw it away anyway OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Solution · We won’t have line-rate optical packet-switching anytime soon - No photonic transistors - No photonic memory buffers · We can do optical circuit switching · Useless for most commercial networks - Doesn’t aggregate for economy of scale - But bulk P 2 P purchases can be extremely attractive · A good match for DOE high-performance science applications which require both bandwidth and precision - Petabyte file transmission in 2 -3 years - Tera-scale interactive visualization ASAP - Remote real-time instrument control now OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Ultra. Net Research Testbed · Build a sparse, lambda-switching, dedicated channel-provisioning testbed · Connect hubs close to DOE’s largest Science users (but let the user labs pay last-mile costs) · Provide an evolving matrix of switching capabilities · Separately fund research projects (e. g. , highperformance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

The Resources: · Off hours capacity from Esnet - Probably 2 x OC 48 between Sunnyvale and Chicago in the northern route · Dedicated Lambdas on NLR - Two 10 G lambdas between Chicago and Sunnyvale - Possibly more in year two or three · Two dedicated lambdas on the ORNL Chicago-Atlanta connector OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Resources Cont. · A progression of switching technologies - Start with Ciena, Cisco, or Sycamore - Migrate to Calient, Glimmerglass all-optical or a hybrid · Some Local Storage - Logistical Storage Depot (prior to local application development) · Progression of experimental point-to-point transport technologies (plus legacy) - Fiber channel - Infiniband · Migrate to the Production ESnet environment OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

CERN Science Ultra. Net Phase I Seattle PNNL 2 x 10 Gbps LBL Star. Light NERSC BNL 2 x 10 Gbps SLAC Sunnyvale ANL Cal. Tech FNAL ORNL JLab SOX Research Exchange Point DOE University Partners DOE National Lab OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Access via production networks Research 10 Gbps lambda

The Physical View From 50, 000 feet ESnet CERN etc. NLR ORNL Connector OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Network Overview CI OC 192 10 Gb. E OC 192 2 x OC 192 CHI – ATL 2 x OC 192 Tx ETF CHI – ORNL Ultranet 10 Gb. E OC 192 OLA - 3 OLA 1 10 Gb. E OC 192 OLA -1 Oakridge CI Gb. E / 10 Gb. E / OC 102 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 10 Gb. E OC 192 20 x Gig-E? ? ? Knoxville Atalnta Acworth Dalton Chattanooga 10 Gb. E OC 192 SOADM conversion Memphis Stevenson Christiana Nashville Bowling Green Upton Louisville Seymour Indianapolis Crawfordsville Monon Schneider Chicago Intro 10 GLAN PHY – CD 10 Gb. E OC 192

The Logical View OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Schedule · Oak Ridge Connector to be up in June - Fall-back circuits via NLR if necessary · Chicago-Sunnyvale paced by NLR - PNNL fiber schedule will pace their connection · Expect first user-traffic in June OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Future Plans · ORNL is half the team building CHEETAH - Nagi Rao and Malathi Veeraraghavan are PI’s Funded by NSF Will Connect ORNL, Atlanta, NC State, possibly CUNY Explicitly developing signaling protocols · Want to partner with HOPI · …and all of you OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Thank You OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY