ANI ESnet 5 Joe Metzger Network Engineer ESnet

ANI & ESnet 5 Joe Metzger, Network Engineer ESnet Network Engineering Group ESCC Fairbanks, AK July 13, 2011

Where are we going: • Understanding the Capabilities • Assets • Services • Building the networks • Dark Fiber Network • Optical Transport Network • ANI Phase 1 • ANI Phase 2 • Transition to ESnet 5 • Timeline 9/6/2021 Lawrence Berkeley National Laboratory 2 U. S. Department of Energy | Office of Science

Assets: 3 Separate Networks! Dark Fiber Network • Long Haul fibers available for ANI Testbed & disruptive research uses for the near future • May be used for the next generation network when we outgrow the Optical Transport Network • Provides point to point wave services for IP, SDN & Testbed purposes • Up to 44 100 G waves on the backbone • Internet 2 DF, shared chassis & amps, ESnet cards for our waves • Up to 88 100 G waves on the CHI and SFBay Express • ESnet equipment on ESnet DF, managed by Internet 2 & Level 3 • Up to 61 100 G waves on the ORNL Express • Level 3 owned equipment on Level 3 DF 100 G Routed network • Engineered to support science applications at 100 G speeds 9/6/2021 Lawrence Berkeley National Laboratory 3 U. S. Department of Energy | Office of Science

Services • Routed Network • Traditional IP Services • Current OSCARS/SDN services and on-demand circuits • Minutes to provision, services last minutes to years • Optical Transport Network • Point to Point long-lived circuits • Weeks to months to provision, services last until components redeployed • Provisioning circuits will require physically deploying transponders • Limited SONET like re-routing around failures in the express networks. Not likely in the WAN due to the regen requirements. • Dark Fiber Network • Provisioning services will require equipment at end-points, regen and amp huts, installing transport gear, etc. This will take months. 9/6/2021 Lawrence Berkeley National Laboratory 4 U. S. Department of Energy | Office of Science

Dark Fiber Network 9/6/2021 Lawrence Berkeley National Laboratory 5 U. S. Department of Energy | Office of Science

Optical Transport Network 9/6/2021 Lawrence Berkeley National Laboratory 6 U. S. Department of Energy | Office of Science

Optical Nodes Add-Drop sites will have 1 or more Ciena 6500 chassis. Waves can be regenerated at adddrop sites if required. Regen required every 900 – 1600 km depending on transponders & power budgets. SFBay & Chicago express nodes at sites will have: • • 1 32 slot Ciena 6500 Chassis Sentry PDU Rectifier Plus all the current ESnet equipment. 9/6/2021 Lawrence Berkeley National Laboratory 7 U. S. Department of Energy | Office of Science

ANI Phase 1 9/6/2021 Lawrence Berkeley National Laboratory 8 U. S. Department of Energy | Office of Science

ANI Phase 2 9/6/2021 Lawrence Berkeley National Laboratory 9 U. S. Department of Energy | Office of Science

ESnet 5 The general plan is: • Build ANI Phase 1, test it, declare success! • Build ANI Phase 2 • Merge ANI Phase 2 and ESnet 4 to create ESnet 5 But, it probably will not happen quite this cleanly • SC 11 • Space & Power constraints • Some users will want to utilize ANI for traffic that is currently using ESnet 4. 9/6/2021 Lawrence Berkeley National Laboratory 10 U. S. Department of Energy | Office of Science

ESnet 5 Conceptual design is done, detailed design is not finished. Significant points from ESnet 5 Conceptual Design • It will look a lot like the ANI Phase-2, with the addition of the rest of the sites & exchanges. • We will be reducing the number of routers in the network compared to ESnet 4 Only 1 100 G router in a hub • No 100 G routers at the sites (temporary exception for ANI Testbed) • • Many hubs will have only 1 router. • Configuring the SFBay Express Routed Links in a dual star instead of a shared ring Trade off larger number of shared links for smaller number of dedicated links • Will be augmented as necessary with additional point-to-point capacity • May be able to use optical protection for individual circuits • Sites can decide how much of their circuits are available for SDN. • 9/6/2021 Lawrence Berkeley National Laboratory 11 U. S. Department of Energy | Office of Science

SFBay Express Optical Network 9/6/2021 Lawrence Berkeley National Laboratory 12 U. S. Department of Energy | Office of Science

SFBay Express Routed Network Concept Evolution 9/6/2021 Lawrence Berkeley National Laboratory 13 U. S. Department of Energy | Office of Science

ESnet 5 Some of the open questions: • 100 G Ciena transponders & 100 G pluggable optics (CFP) issues. • What is the optimal configuration for ESnet equipment on a site that is on a metro ring? • What is the best layer for protection & reliability? • Do we want 0, 1 or 2 ESnet routers at a site? • How valuable are slower line-rate backup paths? • If a site has a 100 G connection, is a divers 10 G backup connection useful? • How should we manage 100 G traffic so it doesn’t saturate 10 G links? • Who should be managing that? • When will the fiber builds be completed? • Budget Issues 9/6/2021 Lawrence Berkeley National Laboratory 14 U. S. Department of Energy | Office of Science

Target Timeline 1. Transport Contract signed in June (Dark fiber + Optical Networks) 2. Router Contract should be signed in July 3. On-Site meetings with the Phase 1 ANI Sites starts next week to deal with dark fiber installation issues. On-Site meetings with other SFBay & Chicago Express sites will follow quickly. 4. The portion of the Optical Transport network required for the ANI Phase 1 will be ready for provisioning wave services by end of August 2011 5. ANI Phase 1 completed ASAP & before end of December 2011! 6. The rest of optical transport network completed by July 2012 7. ANI Phase 2 completed by July 2012 8. ESnet 5 transition complete before November 2012 9/6/2021 Lawrence Berkeley National Laboratory 15 U. S. Department of Energy | Office of Science