NLT 1 Expectations findings and innovation Bas Kreukniet

NL-T 1 Expectations, findings, and innovation Bas Kreukniet, Sr Network Specialist at SURFSARA Geneva Workshop 10 Februari 2014

Outline 1. Expectations from NL-T 1 grid administrators 2. Findings while connecting to the LHCONE 3. Innovation: Ethernet OAM and NSI

Expectations from NL-T 1 administrators Advice from grid administrators NL-T 1: • Bulk data with simple applications (grid-FTP). “Keep it simple” • Network provisioning from application is considered “complex” • Don’t rush to merge LHCOPN with LHCONE

NL-T 1 connected to LHCONE since 20 Jan 2014

BGP routing starts at the connected site. BGP routing for T 1’s: Not only a “NREN thing” – it already starts at your organisation! Focus on some BGP topics for connecting sites to LHC networks

LHCOPN connectivity

LHCONE connectivity LHCONE. inet. 0: 133 destinations, 133 routes (133 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both NL-T 1 AS 1162 GÉANT AS 20965 LHCONE-RS CERN AS 20641 128. 142. 0. 0/16 *[BGP/170] 1 w 2 d 04: 07: 47, MED 120, localpref 100 AS path: 20965 20641 513 I > to 62. 40. 126. 161 via xe-4/0/1. 2012 CERN AS 513

Preferred route: LHCOPN, LHCONE or Internet • • primary: direct T 1–T 1 over LHCOPN secondary: T 1–T 1 over LHCOPN via another T 1 tertiary: LHCONE quaternary: Internet

Preferred route: route (a)symmetry If everyone makes this choice, all connections are symmetric. But: sites may have different preferences: ISP-A 10 Gb/s site 1 10 Gb/s 100 Gb/s ISP-B site 2

Tie-breaker between LHCOPN and LHCONE CNAF routers at NL-T 1 LHCOPN 131. 154. 128. 0/17 *[BGP/170] 14: 58: 18, localpref 100 AS path: 34878 137 I LHCONE 131. 154. 128. 0/17 *[BGP/170] 1 w 2 d 04: 23: 40, MED 120, localpref 100 AS path: 20965 137 I

LHCOPN / LHCONE route preference BNL routes at NL-T 1 LHCOPN 130. 199. 185. 0/24 *[BGP/170] 2 w 5 d 09: 35: 43, MED 10, localpref 100 AS path: 513 43 I > to 192. 166. 73 via xe-1/1/0. 0 [BGP/170] 3 d 01: 15: 22, MED 51, localpref 100 AS path: 39590 513 43 I > to 109. 105. 124. 17 via xe-2/1/0. 0 LHCONE 130. 199. 185. 0/24 *[BGP/170] 1 w 2 d 04: 27: 43, MED 120, localpref 100 AS path: 20965 293 43 I > to 62. 40. 126. 161 via xe-4/0/1. 2012

Problems and concerns regarding BGP • A site advertising his routes has no control who to send a route to. At best they can give hints with BGP communities. • The site receiving a route decides which route to accept and how to accept.

BGP in LHC networks Only As is not enough. More info needed. community for origin of a route T 1/T 2? Or even better: -site connected to LHCOPN -site connected to LHCONE - Site connected to both: Type A “prefer LHCONE for this route” Type B “prefer LHCOPN for this route” - Specials: dedicated link between two (T 1) sites. (“VPN”or “private link”)

Route Preference Solutions • Idea: Tag routes with two types of communities: • One for origin or source • One for destinations • See also: BGP hinting by Martin Sweeny (Indiana U) • BGP Always-compare-MED always on. We sometimes add metrics on incoming routes.

Operational Issues • We received routes over LHC from a site, but traffic we send is blackholed by that site • The site was still reachable over the Internet. • This happened to us twice recently: on LHCOPN (accidental route redistribution) and LHCONE (incoming IP filter). • Configuration errors will be made (we’re also just human) • LHCOPN Link NL-T 1 – TRIUMF link is still unstable • 31 outages last 4 months Monitoring remains important!

LHCONE and LHCOPN layer 2 monitoring Ethernet OAM monitoring or Layer 2 monitoring NIKHEF, TRIUMF to participate as Measurement Points Looking for T 1’s and T 2 to participate Advantages: • Layer 2 keep-alive, ping and traceroute • Interdomain, intervendor solution • L 2 devices can be made visible • unidirectional fibercuts can be signalled

LHCONE Innovation NL-T 1 likes to partcipate in NSI for LHCONE NSI experiences so far: • Use-case: Life Science Grid (LSG) in Holland makes use of NSI since autumn 2013. • Freek contributed to standard • Sander wrote NSI client and implemented it for “Cloud Bypassing” in the Life Science Grid.

Cloud Bypassing on Life Science Grid (LSG) Compute clusters at ± 10 locations in the Netherlands

Cloud Bypassing on Life Science Grid (LSG) Offloading campus networks (some only have 1 Gb/s Internet)

LHCONE Innovation: Offloading is cheaper Internet full routing (incl backup): € 8 k – € 10 k per month for 10 Gb/s LHCONE or dynamic lightpath: € 2 k – 3 k per month for 10 Gb/s

Questions? Erik Ruiter Bas Kreukniet Diederik Vandevenne Sander Boele Farhad Davani Freek Dijkstra