LHCONE Implementation in America and Example Site Configurations

LHCONE Implementation in America and Example Site Configurations William Johnston Senior Scientist and Advisor ESnet Lawrence Berkeley National Laboratory Asia Tier Center Forum KISTI, Daejeon, South Korea September 22 -24, 2015

LHCONE in America • Networking in the U. S. is structured somewhat differently than it is in Europe. • The U. S. has many independent networks that have informal arrangements with each other to provide a coherent R&E national network • The potential for fragmentation was recognized in the mid-1990 s and NSF funded the construction of an initial set of “Giga. Po. Ps” (gigabit point-of-presence) that were open exchange points where networks could interconnect in a policy-free environment • Today the Giga. Po. Ps or “exchange points” are an integral part of the U. S. R&E Internet – Star. Light, MAN LAN, WIX, SOX, PNG, and many others • The Giga. Po. Ps play a key role in the U. S. LHCONE environment by providing the inter-VRF connections 2 6/5/2021

LHCONE in America • Although LHCONE is frequently represented just by the connections between NSP and site VRFs, reality is quite different CANARIE Canada UVic SCINET (UTor) TRIUMF-T 1 Mc. Gill Sim. Fra. U NORDUnet CERN ESnet USA FNAL-T 1 PNNL-T 1 SLAC UCSD UNL AGLT 2 MSU Caltech IU (MWT 2) 6/5/2021 CERN UChi (MWT 2) UIUC (MWT 2) MIT UWisc NE Internet 2 USA 3 AGLT 2 UM UFlorida Pur. U Vanderbilt GEANT BNL-T 1 GLakes Harvard So. W

LHCONE in America • The reality is that virtually all of these bilateral connections are mediated through an exchange point CANARIE Canada UVic UTor TRIUMF-T 1 Mc. Gill SINET+ GÉANT, MAN LAN Sim. Fra. UUBC Star. Light ESnet Star. Light Pac. Wave Los Angles SIne t Caltech Internet 2 ESnet SIINET Caltech KRE ONE CAN T ARI E GÉANT Ne (Am ther. L ste ight rda m) CST Net SInet Multipoint exchange for LHCONE rt Internet 2, CERN, MIT, UChi, UFla, UOak, Vand, , GÉANT (all of NRENS and sites) and Brazilian sites), ESnet, NORDUNet, RUNNNET (HEPNET, GRIDPNPI, IHEP) CANARIE fu nk Fr a N T/ ET N ÉA va ne Ge T/ SI G IU (MWT UIUC 2) (MWT 2) Internet 2 USA MIT Harvard AN KR AS EO GC NE T UCS D CIC Omni. Po. P (Chicago) MAN LAN (New York) AGLT 2 UM n SI t DUn e R , Harvard Pur. U NO et et, n, T, S Vanderbilt h N a. Et NE So. W m. P GÉA MAN A UFlorida ia v via LAN SP TTA P AN NOR DUnet AS GC SINET to Internet 2 Intern e SIIN t 2 E Calt T UCSech D Pacific. Wave (distributed exchange) CAN ARIENOR DUN et UWisc AGLT 2 UNL MSU UChi (MWT 2) an CA All s NARIE pres ites e MA nt at N mult LAN ipo VLA int N , Taiw CAN ARIE Kurchatov Internet 2 FNAL-T 1 SLAC ASGC ESnet USA Internet 2 UNL, Wisc, AGLT 2, UChi, ESnet AS GC BNL-T 1 KISTI PNNL-T 1 C UIU IU 6/5/2021 NA RIE ndon t, Lo ESne st. t, Am ESne neva t, Ge ESne UNL, Wisc, hi 4 CA AGLT 2, UC • Some of the exchange points set up multipoint exchanges for LHCONE VRF cross connections (via either a dedicated physical switch or a virtual switch) KRE ONE T Starlight (Chicago) CSTNet CUDI • Some of the exchange points are distributed in that cross domain connections can be made at different physical locations Pac. Wave / PNWG (Seattle) GÉANT, Paris GÉ WIX (Washington) SOX Atlantic. Wave (distributed exchange) AMPATH NAP of Americas (Miami)

LHCONE Basic Site Architecture Example-I • There a number of possible configurations and relationships between sites and the Network Service Provider runs the LHCONE VRF LHCONE NSP Routed IP Service VRF 1 VLAN 2 LHCO NE Traffic VLAN 1 Genera l IP Traffic • Sites must provide a separate subnet (and address space) for LHC resources VRF 0 Security scan – Separate routing instance for LHCONE – LHCONE subnet address block published to CERN – Direct access by LHCONE (and LHCOPN) to site LHC resources • Very much like a Science. DMZ – See fasterdata. es. net 5 6/5/2021 Border Router (2 VRFs) Campus Core the LHCONE VLAN connects directly to the LHCONE subnet switch/route r LHC Tier-x Center Thanks to Shawn Mc. Kee, U. Mich.

LHCONE Basic Site Architecture Example-II • Most big US sites, as well as others, run their own VRF because they want to have control over the LHCONE routing and have the ability to easily impose policy on the incoming traffic NSP LHCONE Routed IP Service VLAN 2 LHCO NE Traffic VLAN 1 Genera l IP Traffic VRF 1 VRF 0 Security scan • Note the for performance reasons the LHCONE subnet is frequently outside the site firewall Campus Core – This is “reasonable” 6 because the LHCONE sites agree to a common security policy and are all in the same science community (LHC) 6/5/2021 Border Router (2 VRFs) LHC Tier-x Center Thanks to Shawn Mc. Kee, U. Mich.

LHCONE Site Architecture Example-IIa NSP • A variant of the basic architecture is that LHCONE – a policy is implemented that lets data from the campus move directly to the LHC systems – however, traffic to campus from the LHC systems is security checked as through it were external traffic • Sometimes called a “diode” approach, this allows for high-speed data loading of LHC servers from campus, but screens all LHC Tier -x traffic because this environment is open to non-campus users 7 6/5/2021 VLAN 2 LHCO NE Traffic VRF 1 Routed IP Service VLAN 1 Genera l IP Traffic VRF 0 Border Router (2 VRFs) Security scan Campus Core Policy Based Routin g LHC Tier-x Center

LHCONE Large Site Architecture Example • Policy Based Routing allows for sitespecific decisions on what LHCONE sites will have direct access to the site LHC resources LHCONE • This is a simplified diagram because such a site will probably have redundant border routers and redundant fiber to their upstream 8 6/5/2021 provider Routed IP Service Pt-2 -Pt Circuits – E. g. have they agreed to the LHCONE security policy? LHCONE VRF 1 VRF 0 Border Router (2 VRFs) Security scan Policy Based Routin g Campus Core CMS Tier-1 Center Thanks to Phil De. Mar, Fermilab and Bruno Hoeft, KIT

LHCONE Site Architecture Example • PBR is just one way to manage the incoming traffic, ACLs could be used, and in the (near) future, Open. Flow traffic management will likely be used But the net result is mostly the same: • Non-conforming traffic is treated differently – In this example, 9 it is routed 6/5/2021 through the LHCONE Routed IP Service Pt-2 -Pt Circuits VRF 1 Policy Based Routin g VRF 0 Policy Route d Traffic Border Router (2 VRFs) Security scan CMS Tier-1 Center Campus Core

LHCONE Site Architecture Example • In this example the nonconforming traffic is returned via the general Internet LHCONE Routed IP Service Pt-2 -Pt Circuits VRF 1 Policy Based Routin g VRF 0 Policy Route d Traffic Security scan CMS Tier-1 Center 10 6/5/2021 Border Router (2 VRFs) Campus Core

LHCONE Site Architecture Example • If another site treats this center’s traffic as non-conforming then that traffic is returned over the general Internet and the site must be able to route that traffic to the LHC resources internally (as illustrated here) • If there is an LHC Tier-2 site that is not connected via LHCONE, then that traffic just 11 looks 6/5/2021 like LHCONE LHC site without LHCONE connection Routed IP Service Pt-2 -Pt Circuits VRF 1 Policy Based Routin g VRF 0 Policy Route d Traffic Border Router (2 VRFs) Security scan Campus Core CMS Tier-1 Center • This traffic must also be able to reach the LHCONE resources

END ØEND

6/5/2021 13

14