Implementing and Maintaining an ISP Backbone Kevin Butler

Tier 1 ISP Backbones • Comprise some of the world’s largest IP networks • Tier 1 companies include Sprint, AT&T, PSINet • UUNET has the world’s largest IP data network (by number of POPs), presence on five continents (North and South America, Europe, Asia, Australia)

Service Level Agreements • SLAs are an important and prestigious tool in attracting and maintaining customers • Comprised of uptime guarantees and bounds on latency through various geographic regions • most ISPs currently have latency < 65 ms monthly average between regional hubs in the US

Current SLA latency times • Looking at the North American Backbone over past 24 hours (ICMP tests) – UUNET: 64. 9 ms – Sprint. Link: 69. 3 ms – AT&T: 68. 7 ms – Cable & Wireless: 60. 8 ms – PSINet: 80 ms source: http: //ratings. miq. net

Supporting the Customer • Quality and expertise of first-line customer support varies wildly between companies • depending on size, geographic location and company focus, some front-line support teams outsourced to third parties • some in-house high level support teams have skills equivalent or superior to NOCs

Network Operations Centres • Generally the teams concerned with backbone maintenance and support • trend towards consolidation into “Super. NOCs” (eg. one for Americas, one for Europe) • specialisation within NOC for product support (eg. dial, VPN, backbone NOCs)

NOC Tools • NOCOL - Network Operations Centre On Line (freeware UNIX) • Mediahouse monitoring (mainly web) • Micromuse Netcool - used by World. Com, PSINet, BT

Some Circuit Terminology • DS-1 = 1. 544 Mbps, refers to “digital signal”, the actual physical layer component • Often used interchangeably with “T 1”, referring to the carrier on the line • DS-3 (T 3) = 44. 736 Mbps or 28 DS-1 s • PRI: “primary rate interface”, equivalent to a DS-1 • BRI: basic rate interface, made up of 2 B (bearer) channels and 1 D channel: B channel is 56/64 kbps (depending on switching limitations), 23 B + 1 64 kbps D channel make a PRI (each B channel is a DS-0 circuit) • Note: 24 DS-0 = 1. 536 Mbps – remainder of bandwidth comes as a synchronizing Frame bit after a byte transferred from all 24 channels (so this is bit 193)

Optical Carrier • OC-x rates based on multiplexing SONET streams • SONET – synchronous optical network: defines a standard optical TDM system with common standards and compatibility across continents (devised at Bellcore) – Europe uses SDH, very similar to SONET • OC-3 = 155. 54 Mbps, commonly goes up in multiples of four in North America and Europe (OC-12 = 622 Mbps, OC 48 ~ 2. 5 Gbps, OC-192 ~ 10 Gbps)

Dial Access • Dial is a major selling point, especially with customers who travel a lot or are their own ISPs • connections made through a dial concentrating unit eg. Ascend (Lucent) MAX TNT, which can support up to 720 concurrent callers • back-end is a DS-3 into a backbone router, routers advertised by an IGP (eg. RIP)

Dial-Related Technologies • COBRA (Central Office Based Remote Access) allow building of virtual POPs by backhauling PRIs • RADIUS (Remote Authentication Dial In User Service) – authenticates and can provide some routing and netblock information about customer logging in

Integrated Services Digital Network • ISDN customers authenticate by RADIUS similar to dial users • Most customers use BRI (2 B channels for 128 kbps data rate) • underlying architecture similar but dial equipment often administrated differently • ISDN maintained within same AS as backbone whereas dial often in its own AS

DS-1 and high-speed access • Customer connections usually multiplexed, come into DSU (data service unit) as a channelised DS-3 • gateway routers on ISP side usually Cisco 7500 series, increasingly using Cisco 12000 • customers connect using Cisco 1604, 2621, some 3600 series, very large customers use 7500 series routers

Gateway Routers • obtain routes from customers usually statically, but sometimes by BGP • usually run link-state IGP within AS (eg. OSPF, IS-IS) • Cisco 7513 backplanes 1. 8 Gbps while 12008 does 40 Gbps

Where does traffic go from here? • Most ISPs have two levels of networks above the access router • Metropolitan networks aggregateway traffic, generally city-wide if multiple points of presence (POPs) in city • transit networks aggregate metro network’s traffic, responsible for inter-city transport

The Big Picture TR TA TA TR TRANSIT XR XR HA DR HA GW METRO EDGE HA DR HA GW

POPs and NAPs as real estate • Often located in the centre of cities (Ameritech NAP in Chicago, right) • 60 Hudson St, NYC is a “telco hotel”, large number of telecoms companies have equipment there • Industrial buildings (because of high HVAC use) and often nondescript (both for cost and security reasons)

ATM Switches • Terminate long-haul OC-12, OC-48 circuits and metro rings • Choice of vendor contingent on ISP, commonly Newbridge, Fore Systems (ASX 1000 and ASX-4000)

Example of an ATM interface TR 1. EG 1: interface ATM 2/0 description To HA 13. BLAH 1 3 C 1 atm vc-per-vp 512 atm pvc 16 0 16 ilmi ! interface ATM 2/0. 195 point-to-point description To XR 1. BLAH 1 ATM 6/0 ip address 146. 188. 200. 98 255. 252 ip router isis Net-Backbone atm pvc 195 0 195 aal 5 snap clns router isis Net-Backbone

Tying it all Together • ATM devices perform switching functions at layer two level • Within regional areas, routers use intradomain routing protocols • To communicate with other regions and across peering points, an inter-domain routing protocol is used

Slash Notation • Subnet masks can be an unwieldy thing to deal with, eg. 255. 240 • Slash notation simplifies this: the number after the slash refers to the number of bits to be ANDed to create the network identifier – 192. 168. 1. 0 255. 0 = 192. 168. 1. 0/24 • Nifty trick: number of hosts in a netblock easy to determine with slash notation - # usable hosts in /x = 2^(32 -x) – 2 • Therefore, there are 256 addresses in a /24, 254 usable

Routing Protocols • Intra-domain (IGPs) – Distance-vector (RIP, IGRP) – Link-state (OSPF, IS-IS, EIGRP) • Inter-domain (EGPs) – Path-vector: BGP • Routes by number of hops between autonomous systems, hence uses a vector comprised of AS sequence numbers instead of next IP address

Autonomous Systems • An autonomous system (AS) is a group of routers with a single routing policy, running under a single administration • Different ISPs, and large companies, can have their own AS number • Where to get a number? In North America, ARIN (American Registry for Internet Numbers), in Europe, RIPE (Réseaux IP Européens), in Asia APNIC (Asia-Pacific Network Information Centre) – also the places for getting IP addresses

Implementation of BGP • BGP runs between autonomous systems and peers, as well as multi-homed customers • monolithic AS broken up into BGP confederations for ease of work • Why BGP? Policies can be defined and routes controlled to a highly customisable degree using access lists and route maps – one can choose what routes to distribute to which neighbours • BGP can run inside an AS – internal (IBGP) carries transit traffic through the AS (like an Interstate through a county)

BGP Communities are destinations that share common attributes (eg. through access-list filters) BGP table version is 23718690, local router ID is 205. 150. 242. 2 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i 24. 64. 0. 0/19 *>i 24. 64. 0. 0/14 *>i 24. 64. 32. 0/19 *>i 24. 64. 96. 0/19 *>i 24. 64. 192. 0/19 *>i 24. 64. 224. 0/19 *>i 24. 65. 0. 0/19 *>i 24. 65. 96. 0/19 *>i 24. 65. 128. 0/19 Next Hop 198. 133. 49. 7 198. 133. 49. 7 Metric Loc. Prf Weight Path 100 0 6327 6172 100 0 6327 i 100 0 6327 6172 100 0 6327 6172 i i i i i

Advantages of BGP for User • Allows for load-sharing and redundancy • routes can be biased through AS path prepending (adding the same AS number to a route multiple times to make it a less favourable route to take) • requirement is high-quality router with close to 100% uptime to avoid connection flaps and subsequent route dampening (BGP gets annoyed if connections go up and down frequently and will penalise the offending network)

Common Customer Issues • Static routes on backbone - often difficult to spot, can cause very strange routing results (very conducive to routing loops) • pull-up routes for netblocks smaller than /24, required to avoid BGP dampening (smaller customers tend to reset their equipment more often) • BGP recalculations - if done on a transit router, entire backbone segments can experience outages (tables are huge, currently over 103, 000 prefixes in table)

Customer Requirements of the Backbone • Redundancy - networks are redundant but card failures can take down whole routers • physical connection to POP from customer is SPF • low latency - massive increases in demand on backbone makes this difficult • over $2 million a day spent on global backbone upgrades

DSL: low cost, high speed • DSL might phase out ISDN connections • difficult to troubleshoot from network standpoint • connections pass through telco’s frame or ATM cloud between DSLAM (DSL access multiplexor – separates voice and data traffic by frequency) and VR • Red. Back SMS (Subscriber Management System) 1000 commonly used as VR, though currently the SMS 10000 is the largest “carrier-class” routing switch, can take in 24 OC-12 s)

Red. Back SMS 1000 • Supports up to 4000 sessions • OC-3 out to metro network • traffic-shaping accomplished with profiles atm profile samplecust counters shaping vbr-nrt pcr 1000 cdvt 100 scr 100 bt 10

Increasing Capacity • Backbone capacity increasing at a huge rate • Traffic engineering combined with high backplane becoming increasingly important • many ISPs turning to Juniper routers • UUNET rolled out production OC-192 c with Juniper M 160 running MPLS

Juniper Routers • Specialises in huge routers (M 160 backplanes 160 Gbps) • JUNOS supports MPLS and RSVP isis { interface all; } ospf { area 0. 0 { interface so-0/0/0 { metric 15; retransmit-interval 10; hello-interval 5; } } } [edit]

Network Abuse • Spam-killing – looking at SMTP header for IP address, null-routing it • Open relay detection – ORBS et. al. • DDo. S attacks can be very detrimental to backbone (even causing switch crashes) • Combated by rate-limiting ICMP on routers • Most effective defense is community-wide egress filtering; requires co-operation throughout the Internet

Network Challenges eg. Canada • Geographically, population resides in virtually a straight line across the south • major focus is on southbound capacity to the US • CRTC regulations on telcos create different arrangements • heterogeneous network to the US, integration a big issue

Costs • Network equipment not cheap: a Cisco GSR can cost upwards of a quarter million dollars • Fibre and transceivers can be expensive to lay ($100 K/mile near rail, over $300 K/mile in the city) • Interesting note: Sprint grew its all fibre network quickly because it was laid on railway right-of-way (the SPR in Sprint initially stood for Southern Pacific Railway) • Costs for backbone access? Currently ~ $1300 CDN + local loop cost for burstable 128 k T 1, up to ~ $50 K CDN for a full T 3, much more for OC 3+ (USD costs similar)

Questions? • Anything I can clarify or expand on. . . • Thank you!