Deutsches Forschungsnetz Rollout of a National Research Network

Deutsches Forschungsnetz

Roll-out of a National Research Network a Field Report Spring Internet 2 Member Meeting Martin Wilhelm German National Research Network DFN wilhelm@dfn. de

Why a new network for DFN ? • Contract with operator (TSI, subsidiary of Deutsche Telekom) ended at the end of 2005 • Market situation had changed significantly (availability of dark fiber, lower prices) • Requirements of users had changed a lot. In particular increased demand for – high capacities between relatively few sites – improved reliability and availability – more capacity for same / lower price 3

The „Old“ Network G-Wi. N 4

Pre-Tender: Market Survey • Market Survey started in early 2004 • Constraint: No budget for investments, so: – asking for offers of DWDM-services (no purchase of equipment) – asking for offers of rental of DF (no IRUs with upfront payment) • Results of Survey: – there is a market for DF in Germany (not the case when tendering for the „old“ Network) – a number of competitors can offer DWDM services at reasonable prices 5

Preparing Call for Tender • The top 60 sites (by volume) were defined as potential end points for dark fiber (DF) as well as for Managed Wave Lengths (MWL), resulting in a 60 x 60 matrix • Bidders were asked to offer options for DF and / or MWL between any of two sites • In another lot bidders were asked to offer DWDM-services on DF provided by DFN • In a 4 th lot offers for network supervision were solicited (24 * 7 * 365) 6

Results of Tender (1) • DFs were offered between a significant number of DFN sites at reasonable prices • Comparing prices for MWL and cost for DF plus DWDM-services showed in many cases economic benefits for DF-solution from 2 nd wavelength on (surprise, surprise. . . ) • For a few areas there was either no offer for DF or offers were economically not reasonable (the German DF badlands ; -)) 7

Results of Tender (2) • As a nice side effect, offers of DF could be arranged to build a number of rings. In total X -Wi. N now spans across approx. 5, 500 fiber kilometers (approx. 3440 miles) • These rings then were complemented by a number of MWLs • Increased number of nodes reduced number of access lines cost savings 8

X-Wi. N-Topology KOP (Fibers and Managed Wavelengths) KIE Fiber KPN AWI Fiber GL GRE ROS HAM EWE DES Fiber GC BRE Fiber DFN ENS BIE MUE DUI BRA KAS FZJ AAC POT HAN Managed WL GOE MAR BIR GIE FRA MAG STB ZIB ADH DRE JEN CHE ILM BAY ESF WUE HEI FZK FFO HUB LEI GSI SAA TUB ERL REG STU 46 Core Sites (+ 3 CBF) • 31 connected by Fiber • 10 connected by Managed WL • 9 connected by fiber and MWL KEH Richtung Basel BAS AUG GAR As of 18. 01. 2006 9

Another topological view AWI AN 115. 00 BRE HAN BIE 139. 35 145. 0 MUE 141. 6 H 138. 00 AM J K 76. 88 L 3 CHE AJ 107. 0 Z 39 Y 16 AC POT 230. 4 32 E 120. 2 LEI AI 0 102. 2 JEN AG X ZIB 168. 5 D 1 265. 58 D 2 158. 30 THU BAY AF 179. 7 ERL 62. 68 0 79. 71 C ILM 300. 24 AE AH A ESFL-S 1 121. 73 R GSI ADH AA F 69. 7 144. 9 Q 69. 3 27 G AD P FRA HUB 380. 11 FFO MAG 104. AK FZJ 188. 98 10. 00 TUB AL O BIR DES BRA DRE 138. 10 AO KIE N AAC 55. 00 I 156. 9 M DUI 268. 8 W HAM 249. 00 196. 00 132 ROS B 132. 30 S 60. 50 FZK 102 154. 00 BAS V U T 91. 00 HEI REG AP 155, 20 Kehl-S 2 STU 274. 42 GAR operational Under construction 10

DWDM Equipment • Contract (provision of DWDM-services) awarded to Huawei • Includes installation, operation and maintenance of DWDM equipment (carrier grade DWDM) as well as amplification (OLA) • Contract defines SLAs (and penalties) and obligation for pro-active 24*7 monitoring • Access to performance data (read-only) via open non-proprietary interface is part of contract 11

Wavelength Planning • Task 1: Substitute G-Wi. N by X-Wi. N as hassle-free as possible (minimize reconfiguration and relocation of existing routers) – done by (almost) 1: 1 copy of existing SDH-links to wavelengths (STM-16, STM-64) • Task 2: Integrate new DFN sites – this required a number of new routers. Some reconfiguration could not be avoided • Task 3: Optimise wavelength plan after the dust has settled (current work) 12

Roll Out Stage 1 • Co-ordination of providers (DF and DWDM) – exchange of fiber info (lengths of fibre spans, attenuation of spans, location and access regulations of amplifier huts, contact details as phone, fax, e-mail of hotlines, . . . ) • Preparation of DFN-sites – creating space for additional equipment (DWDM) at „old“ sites – preparing grounds (sometimes from scratch) in „new“ sites: installation of cabinets, power, UPS, A/C, patch fields, out-of-band access, . . . 13

Roll Out Stage 2 • Verification of WL routing – Optimal WL routes proposed by Huawei experts – Cross-checked by DFN and verified against IP routing • Building of the (in)famous last mile – Almost all DFN-sites are on university / research lab campuses. All (but two) sites have dual connections. Cross-check that all fiber routes are built with full redundancy (sometimes difficult because of unadequate infrastructure or stubborn administrations) 14

Roll Out Stage 3 • Installation of Optical Line Amplifiers (OLA) – Early installation was possible, because all long distance spans of fibers as well as amplifier huts were already in place • Installation of DWDM equipment at DFN sites 15

Some Installation Impressions 16

More Impressions 17

Even more Impressions 18

Roll Out Stage 4 • Link Commissioning – five teams working in parallel to set up basic interworking between adjacent DWDM locations. Since in most cases OLAs were required, this was done step by step: DWDM A -> OLA 1 -> OLA 2 -> DWDM B -> OLA 3 -> DWDM C, then back to set up the other direction. – Detailed planning was necessary to minimize travel time between sites (problem of traveling sales man ; -)). – Average fuel consumption per working day: approx. 70 l per team 19

Roll Out Stage 5 • Service Commissioning – again five teams in parallel working to set up end -to-end wavelengths including the digital interfaces (STM-64, STM-16, 10 GE, 1 GE), fine tuning of lasers etc. – again quite a bit of travel was involved, although remote set up and calibration was possible at this stage 20

Roll Out Stage 6 • Switch over from „old“ G-Wi. N SDH-links to „new“ X-Wi. N wavelengths • The first approx. 10 links could be set up in parallel to existing SDH-links. This worked well. Good for building confidence ; -) • Since interfaces at all of our routers could not simply be doubled, most of the core links had to be „hot swopped“ (no old links in parallel standby). Some nervosity could be noticed. . . 21

Timeframe was a bit tight. . . • The only thing we were sure of was the cut off date for G-Wi. N: 31 December 2005. . . • Fortunately, remarkably few things went wrong: – 4 or 5 wrong patches, a few faulty patch cables, three faulty boards, one weak laser etc • Last G-Wi. N link switched off on 29/12/05 • A German proverb says: „Luck is with the competent“ : -) • The following slides show time went by 22

Roll out X-Wi. N (15/09/05) KIE DFN-Site AWI Site infrastructure ready ROS HAM DWDM installed GRE DES DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS TUB MAG ZIB JEN ILM BAY GSI WUE HEI SAA CHE GIE FRA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL FZK REG AUG KEH BAS STU GAR 23

Roll out X-Wi. N (26/09/05) KIE DFN-Site AWI Site infrastructure ready ROS HAM DWDM installed GRE DES DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS TUB MAG ZIB JEN ILM BAY GSI WUE HEI SAA CHE GIE FRA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL FZK REG AUG KEH BAS STU GAR 24

Roll out X-Wi. N (29/09/05) KIE DFN-Site AWI Site infrastructure ready ROS HAM DWDM installed GRE DES DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS TUB MAG ZIB JEN ILM BAY GSI WUE HEI SAA CHE GIE FRA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL FZK REG AUG KEH BAS STU GAR 25

Roll out X-Wi. N (02/10/05) KIE DFN-Site AWI Site infrastructure ready ROS HAM DWDM installed GRE DES DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS TUB MAG ZIB JEN ILM BAY GSI WUE HEI SAA CHE GIE FRA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL FZK REG AUG KEH BAS STU GAR 26

Roll out X-Wi. N (13/10/05) KIE DFN-Site AWI Site infrastructure ready ROS HAM DWDM installed GRE DES DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS TUB MAG ZIB JEN ILM BAY GSI WUE HEI SAA CHE GIE FRA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL FZK REG AUG KEH BAS STU GAR 27

Roll out X-Wi. N (31/10/05) KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered ROS HAM DWDM installed GRE BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 28

Roll out X-Wi. N (07/11/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered ROS HAM DWDM installed GRE BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 29

Roll out X-Wi. N (18/11/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered ROS HAM DWDM installed GRE BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 30

Roll out X-Wi. N (21/11/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered ROS HAM DWDM installed GRE BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 31

Roll out X-Wi. N (30/11/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered ROS HAM DWDM installed GRE BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 32

Roll out X-Wi. N (05/12/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered ROS HAM DWDM installed GRE BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 33

Roll out X-Wi. N (14/12/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered BRE DF delivered POT HAN MUE MWL ordered MWL delivered ROS HAM DWDM installed GRE BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 34

Roll out X-Wi. N (19/12/05) KOP KIE DFN-Site AWI Site infrastructure ready DES EWE DF ordered ROS HAM DWDM installed GRE BRE DF delivered POT HAN MUE MWL ordered MWL delivered BIE DUI BRA KAS MAG TUB ZIB JEN GIE FRA CHE ILM BAY GSI WUE HEI SAA ADH DRE MAR BIR AAC HUB GOE LEI FZJ FFO ESF ERL REG FZK AUG KEH BAS STU GAR 35

Today: • Everything now on X-Wi. N • First links of old network were cut off on 2 January 2006 • No operational issues so far. . . 36

Cross Border Fibers CPH AWI KIE HAM ROS DES EWE BRE POT HAN Surfnet (NL) BIE MUE DUI BRA KAS ZIB BAY ESF ERL REG FZK KEH CHE ILM GSI SAA (PL) DRE GIE WUE HEI ADH PSNC GOE JEN FRA FFO HUB MAR BIR AAC (FR) MAG TUB LEI FZJ Renater GRE AUG STU CBF DFN Po. P Dark fibre Wavelength GAR Switch/GARR 37

The Idea behind CBFs • Connectivity between NRENs usually via GÉANT • Cross Border Fibers complement existing GÉANT infrastructure and can add resilience • CBFs have a big potential for savings • Will be part of European infrastructure - not a testbed ! • Challenge: Operational issues. Requires definition of technical, operational, organisational and financial framework 38

Questions. . . ? ? ? ? 39