Network Architecture and IPv 6 Deployment at CERN

Network Architecture and IPv 6 Deployment at CERN CHEP Oct 2013 David. Gutierrez@cern. ch Co-authors: Edoardo. Martelli@cern. ch, Carles. Kishimoto@cern. ch IT/Communication Systems IT Department

Agenda 1. Network Infrastructures 2. LCG Architecture 3. Management 4. IPv 6 Deployment Status 1 IT Department

1. Network Infrastructure 2 IT Department

CORE Network • • Interconnects all infrastructures Extends between Geneva-Wigner General Non-blocking 1 Tbps Purpose IPv 6 Ready External Technical OSPF backbone CORE Security policies LHC Computing Problem isolation Experiments Grid 3 IT Department

General Purpose Technical Network Technical Experiments External CORE LHC Computing Grid • Technical services – Safety, electricity, cooling, … • LHC Operation – Cryogenics, vacuum, … • Industrial systems Devices Switches 8, 073 485 HP Routers IPv 4/IPv 6 Dual Stack 24 NO 4 IT Department

General Purpose Experiments Technical Experiments External CORE LHC Computing Grid • IT/CS provides – Detector Control Network • ALICE • ATLAS • CMS – Data Recording to T 0 • 20 Gbps, up to 50 Gbps • 25 PB/year Devices Switches 6, 111 390 HP Routers IPv 4/IPv 6 Dual Stack 15 NO 5 IT Department

General Purpose LHC Computing Grid Technical Experiments • High Performance Network • 5. 28 Tbps Non-blocking Switching Fabric • WLCG Tier 0 External CORE LHC Computing Grid – Long-term storage – Distribution WLCG – 1. 5 PB/day Devices Switches 8, 902 588 Brocade Routers IPv 4/IPv 6 Dual Stack 13 YES 6 IT Department

General Purpose External Network Technical Experiments External CORE LHC Computing Grid • Public general purpose connections – Full BGP Internet routing table – Geant, CIXP, ISPs • Private WLCG – LHCOPN • 70 Gbps peaks to T 1 – LHCONE Brocade Routers 8 BGP Peerings Aggregated BW IPv 4/IPv 6 Dual Stack 86 232 Gbps YES 7 IT Department

General Purpose Network Technical Experiments External CORE LHC Computing Grid • Desktop computing • Wired and wireless • Central Services – AFS, www, mail, databases Users Buildings Devices 14, 592 650 114, 061 Switches 1, 550 Wi. Fi Access Points 1, 514 HP Routers 100 Brocade Routers IPv 4/IPv 6 Dual Stack 7 YES* 8 IT Department

2. LCG Network Architecture Internet CORE GENEVA WIGNER 9 IT Department

Building block: Service • Network Access modeled as Service • A Service is a broadcast domain – Providing access to end systems – IPv 6/IPv 4 Dual Stack • Features are defined by a Service Type – Jumbo, LACP, tagged, loop protect, accesslist protect, OSPF, VRRP, … • A To. R Switch provides a Service • A Router interconnects multiple Services 10 IT Department

Service Bandwidth • Service capacity depends on Service purpose • Blocking Factor: 2 for CPUs, 5 for Storage CPUs rack Distribution Router Storage rack m x 10 Gbps n x 10 Gbps x 1 Gbps 10 Gbps x 11 IT Department

Scaling the Data Center Rows of Racks Distribution Routers Backbone Routers 100 Gbps links 100 s of 10 Gbps 12 IT Department

Worldwide LCG Border Routers 100 Gbps links ∑ 140 Gbps ∑ 20 Gbps Internet Racks ∑ 12 Gbps Distribution Backbone CORE Network. External Network 13 ∑ 60 Gbps IT Department

Extending the Tier 0 to Wigner Internet CORE Network Racks Distribution ∑ 240 Gbps Backbone 14 IT Department

WLCG Tier 0 Internet Backbone Routers CORE MPLS Racks Distribution Racks Backbone GENEVA WIGNER 15 IT Department

3. Network Management Public Web SOAP Admin Web Configuration Manager Monitoring 16 IT Department

Dyn-A 4 three columns example of a distribution router configuration Racks I think I can fix it! Distribution 17 IT Department

Network Database • • Buildings and cabling infrastructure IPv 4/IPv 6 addressing schemas Services and service features Devices (Servers, VMs, desktops, smartphones, …) Switches Routers/Firewalls Security for the Infrastructure (firewall rules) 18 IT Department

Interfaces to the Network Database Users Public Web Sys Admins SOAP Net Admins First Line Admin Web Configuration Manager Tech Admins Monitoring 19 IT Department

Software-based Network configuration Net Admins First Line Automatic Updates Configuration Manager General Purpose Technical Experiments External CORE LHC Computing Grid Network Services • DNS • DHCP • RADIUS 20 IT Department

4. IPv 6 Deployment Status 21 IT Department

Network Database: Schema and Data IPv 6 Ready 2012 Admin Web: IPv 6 integrated Configuration Manager supports IPv 6 routing Gradual deployment on the routing infrastructure starts The Data Center is Dual-Stack 2013 NTPv 6 and DNSv 6 • Infrastructure is Dual-Stack DHCPv 6 Today • Firewallv 6 automated configuration • User Web and SOAP integrate IPv 6 • Automatic DNS AAAA configuration 22 IT Department

IPv 4 / IPv 6 same portfolio • Identical performance, common tools and services • Dual Stack, dual routing – OSPFv 2/OSPFv 3 – BGP ipv 4 and ipv 6 peers • Service managers decide when ready for IPv 6 • Devices must be registered – SLAAC disabled – RAs: Default Gateway + IPv 6 prefixes no-autoconfig – DHCPv 6 • MAC addresses as DUIDs: painful without RFC 6939 • ISC has helped a lot (βcode implementing classes for ipv 6) • DHCPv 6 clients might not work ‘out of the box’ 23 IT Department

Conclusions • The Network is ready to accommodate the new demands after Long Shutdown 1 • Before Eo 2013, IPv 6 will be fully deployed and available to the CERN community 24 IT Department

Thank you! 25 IT Department

Extra Slides 26 IT Department

Data Centers Power Geneva 3, 500 KW ~900 KW 828 90 Servers 10, 173 ~1, 200 Routers 22 6 100 Gbps ports 60 18 To. R Switches 662 140 1 Gbps ports 22, 776 3, 072 10 Gbps ports 4, 284 528 Racks CERN Wigner 2013 Area ~600, 000 m 2 Buildings 646 Staff and Users 14, 592 Devices Registered 170, 475 L 2 Switching Switches To. R Switching 2726 1 Gbps ports 91230 10 Gbps ports 5656 L 3 Switching Storage Disks Raw disk capacity (Ti. B) Tape Drives Data on Tape (Pi. B) Routers 79, 505 124, 660 161 1 Gbps ports 5976 10 Gbps ports 2248 100 Gbps ports 65 78 Wi. Fi Access Points Devices seen/day 27 1, 514 ~7, 000 IT Department