Exchange Point Technology Nanog 20 Oct 22 24

Exchange Point Technology Nanog 20, Oct 22 -24, 2000 Lane Patterson Member Research Staff <lane@equinix. com>

Intro Framework 2

IX Technology Universe • 10/100 Ethernet, FDDI • ATM OC 3/12 (SAR limitations) • Gig. E • Private Copper/Fiber Cross Connect • SONET ADM/DCS • Frame Relay on POS OC 48/192 • LSR (over POS, Ethernet, …) • 10 Gig. E • Optical Switching (w/ signaling of some sort) 3

IX Connection Models • Layer 2 Non-Broadcast Multi-Access (NBMA): • TE and connection-oriented = more control at the expense of management and operational overhead • Frame Relay or ATM PVCs • MPLS LSPs • Even 802. 1 p/q in a “VLAN per /30” model • Layer 2 Shared Media • FDDI, Ethernet, DPT/SRP • Layer 1: • SONET DCS style: STS-1, OC-N • Wavelength, Dark Fiber • Nailed Up TDM Today; GMPLS tomorrow? 4

IX Applications: Current and Potential • Unicast Peering Aggregation • Private or Public • Multicast Peering Aggregation • Interdomain Interconnection Policy • • Human Enforced Clue Route Servers Interdomain MPLS/GMPLS? Qo. S Brokering? • Not focusing outside of interconnection in this talk (e. g. Stratum 1 servers, other services) 5

What We’re Doing 6

Gigabit Ethernet • The one big Unicast VLAN model • But private VLANs and tagged interfaces OK • Must Mitigate Shared Risk: • IX Policy and Enforcement • Switch Filters and Knobs • Cost, simplicity, and operational advantages over ATM • More traffic control features now on Gig. E lowers the contrast with ATM • MTU a problem? • Lots of “core Internet MTU” debate • But more router data helpful—flow cache packet distributions, frag stats, better analysis of packets that are >1500, etc. • Waiting on vendor support: MTU per VLAN tag • Will be announced as a product • I’m the R&D guy not the product guy—usual non-Nanog channels apply 7

Gig. E Shared Risks • No Brainers: Participants • • ARP: no ip proxy-arp Broadcast: no ip directed-broadcast ICMP Redirects: no ip redirects CDP/other noise: no cdp enable No IGP: passive interface No 3 rd party switches, BPDUs Multicast: keep off unicast exchange VLAN 8

Gig. E Shared Risks (cont’d) • IX-side • STP tuning a must • Upcoming 802. 1 w and STP improvements from today’s 2 -4 second reconvergence to milliseconds • Block BPDUs on non-IX-trunk ports • Layer 2/3 filters per port • Trend alarm bridge tables (# and identity of MACs on each port) • Some controls still needed that are easy • Some controls may not be worth it: • • • Too much maintenance overhead for ISPs Static ARP, MAC per PORT/VLAN Shaping/Policing policies Filter Maintenance If we really want this, go back to NBMA models 9

Future Gig. E Developments • Vendors are active • 10 Gig. E • Other proprietary features for control, scaling • Active 802. 1 projects • Still Need Multicast IX Features • PIM snooping for Port/Group state 10

What We’re Tracking/Testing 11

SONET Cross-Connects • Aggregate private circuits through SONET DCS • Assumes OC 48/192 Ports Channelizable to OC-N (both on Router and DCS) • Recent DCS Products: up to 512 OC 48 down to STS-1 • Worthwhile for co-located routers? • • Already precedent: Florida. MIX Good combo with DWDM into building Extends current SONET aggregation at DS 3 & OC 12 Cost of SONET DCS ports vs. Other Alternatives • Policed Ethernet dot 1 q alternatives • Cost/density/flexibility trade-offs versus continuing to burn DS 3/OC 12 router ports • Will report back on future testing progress • Input is welcome 12

NBMA Migration Paths • Frame Relay at OC 48/OC 192 most promising “ATM Upgrade” today • Mature standards • Switches becoming available • MPLS LSRs need more momentum • Still intra-domain focused and limited set of ISPs • Inter-domain signaling issues • Promising longer term for new inter-provider features • • MPLS/BGP VPN Inter-provider Qo. S Dynamic B/W provisioning: circuit-em, signaling None of these proven yet 13

NBMA Migration Paths (cont’d) • Equinix committed to more MPLS LSR Testing • Current Inter-Domain Models • Non-terminating IX LSR: • Static Labels • RSVP+ERO but no IGP—statics across directly-connected IX /30’s • Terminating IX LSR: • Add CCC-style to above combinations • Need more robust Inter-Domain Signaling models • MBGP NLRI for label exchange a first step • Bilateral policy directly on LSRs or real 3 rd party gateway infrastructure? • Same issues carry over to GMPLS and optical world 14

Optical Switching • Dense numbers of signaled wavelength and fiber cross connects • Testing one OXC vendor near term • Testing open to interested parties • Still early in development • For now, wavelength exchange is just a private cross connect between customer-owned DWDM equipment • Fiber/wavelength density and signaling-driven applications will drive this onto OXCs • Then Bill Norton can do some whiz-bang cost benefit graphs • “Multiservice Exchange” 15

Virtual Device Trends… • …or lack of them • Useful for customer-driven, on-demand provisioning • Today not many real virtual devices, just NMS/OSS integration through secure web front-end. • Future boxes that can push this: • 3 rd party policy brokers • Register policy • Integrate statistics for operation, billing • Admission control in secure signaling domain • Real virtual devices: • Log in to your virtual instance on a shared box • Bandwidth trading systems 16

Going Forward • Large IX Facilities: • Focus on removing space, power, and interconnection constraints from scaling equation • Continued ability to accept new participants • Only limit should be strands*wavelengths into the building/campus • Today, • Public Fabric: 10/1000 Ethernet moving to 10 Gig • Private Fabric: dark fiber cross connects • Under Consideration: SONET cross-connects and aggregation onto OC 12/48/192 17

Last Thoughts • When future cores are 80 wavelengths of OC 192, and migrating to 320 wavelengths of OC 768, what do exchange points need to look like? • Other than more bandwidth, what inter-domain services will take hold? • Will we have any hair left by then? 18