Extreme Networking Achieving Nonstop Network Operation Under Extreme

Extreme Networking Achieving Nonstop Network Operation Under Extreme Operating Conditions Jon Turner Fred Kuhns jst@cs. wustl. edufredk@cs. wustl. edu http: //www. arl. wustl. edu/arl

Motivation n Internet subject to extreme traffic conditions. » correlated user behavior; selfish and/or malicious users n Growing reliance on data networks. » higher expectations for reliability and performance n Design networks for worst-case traffic conditions. » practice constructive paranoia » provide carefully regulated reserved bandwidth services » better queueing mechanisms for traffic isolation » network mechanisms to protect web sites from DDOS » plan for continuous upgrading of network infrastructure n n extensible routers that can adapt to new threats, as they appear Technology progress making extreme defenses practical, without sacrificing performance. 2 - Jonathan Turner - July 31, 2001

Extreme Network Services n Lightweight Flow Setup (LFS) » one-way unicast flow with reserved bandwidth, soft-state » no complex signaling, wire-speed setup, easy to deploy n Network Access Service (NAS) » provides controlled access to LFS » registration/authentication of hosts, users » resource usage data collection for monitoring, accounting n Reserved Tree Service (RTS) » configured, semi-private network infrastructure for information service providers » reserved bandwidth, separate queues for traffic isolation » paced upstream forwarding with source-based queues for isolation and DOS protection 3 - Jonathan Turner - July 31, 2001

Can We Afford Per Flow Processing? If it adds value, absolutely. n Per Flow State n » at $50/MB (fast SRAM), 200 B of flow state = 1 cent » at $1/MB (DRAM), 10 KB of flow state = 1 cent » if used for 2000 hours (avg. of <5% over 5 years), costs 1 mcent per hour to cover cost of both n Per Flow Processing » to enable average of 10 instructions/byte on OC-192, need 12. 5 GIPS 10 i/b enough for header processing n 100 i/b enough for DES encryption n » at $200/GIPS, a 10 Mb/s flow will cost 125 mcents/hour » by 2010, expect to do 100 inst. /byte for 12. 5 mc/h 4 - Jonathan Turner - July 31, 2001

Resource Reservation in Internet? n Bandwidth reservation can provide dramatically better performance for some applications. n Obstacles to resource reservation in Internet. » distaste for signaling protocols » perceived complexity of Int. Serv+RSVP » requires end-to-end deployment » little motivation for service providers n How to get resource reservation in Internet? » keep it simple focus on top priorities - one-way unicast flows n avoid complex signaling - leverage hardware routing mechanisms n » make it useful when only partially deployed » provide motivation for ISPs to deploy it 5 - Jonathan Turner - July 31, 2001

Lightweight Flow Setup n Implicit, one-way, unicast flow reservation. » to setup flow, just send packets - no advance signaling » specify flow rate(s) in packet header (using IP option) » flow detected and route selection triggered as needed » route for flow pinned until flow is released or times out » prefer routes with ample unreserved bandwidth n Stable rate reservation. » allocated independently by routers along path » congested links forward packets as datagrams n n reservation request honored as bandwidth released by other flows Transient rate reservation. » routers allocate bandwidth fairly among competing flows » direct feedback of bottleneck bandwidth to senders 6 - Jonathan Turner - July 31, 2001

IP Option for LFS op identifies flow setup operation - release state - reserve stable rate - reserve transient rate - status report code 8 n - status request - ignore allocated rate requested rate length 8 op. flags 4 4 rate 1 8 rate 2 4 Stable rate fraction updated by routers on path. » may trigger usage-based accounting Status request flags trigger status report. n Alloc. rate stored at last hop router for status gen. n F. P. rates with 4 bit mantissa, 4 bit exponent. n » specify rates from 64 Kb/s to 4 Gb/s , 6% “granularity” 7 - Jonathan Turner - July 31, 2001

Implementing LFS - Input Side Flow Table Route Table Flow Proc. . Flow Processor Flow Table Access Table n If flow table entry present, use stored next hop n If no flow table entry, lookup route & create entry » store selected next hop in flow table entry n At access router » check privileges and record usage in access table » if flow setup not enabled, forward packet as datagram 8 - Jonathan Turner - July 31, 2001

Implementing LFS - Output Side Flow Table Route Table Flow Proc. . Flow Processor Flow Table Access Table n If flow table entry present, use it to find queue, otherwise create an entry & allocate queue. n If stable rate specified, update entry. » keep list of unsatisfied reservation requests to process as bandwidth becomes available n If transient rate, update fair share and pacing rate. 9 - Jonathan Turner - July 31, 2001

Example Application Edge Router Web Site ISP Network Private LAN Web site specifies stable rate in outgoing streaming media packets n Use feedback to adjust sending rate if necessary. n Note: no action required by receivers. n 10 - Jonathan Turner - July 31, 2001

Regulating LFS Usage n Regulate LFS use to ensure availability to users. » user-specific privileges (limit rates, # reserved flows, . . . ) n Record usage for monitoring, accounting. » record reservation periods, rates, # bytes delivered User privilege and usage information stored in host/user database. n Regulation & monitoring at network access points. n » for fixed access, just use physical interface » for roaming access to ISP or corporate network registration protocol executed when host connects to network n IP tunnel for data transfers between host and access point n all data to/from host passes through that point n 11 - Jonathan Turner - July 31, 2001

Reserved Tree Service 70 Mb/s downstream Datagram Forwarding Web Site 100 Mb/s 70 Mb/s Entry-Exit Point upstream Reserved Tree 10 Mb/s 15 Mb/s 10 Mb/s n Reserved tree branches out to locations where users are. n Downstream packets forwarded on-tree, share reserved bandwidth pipes. » last hops use datagram forwarding n Upstream packets paced and kept in source-based queues. 12 - Jonathan Turner - July 31, 2001

Extreme Router Architecture • system mgmt. • route table cfg. • setup for non-LFS flows Control Processor Scalable switch fabric Switch Fabric Dist. Q. Ctl. Output Port Proc. Flow Lookup . . . Input Dist. Q. Ctl. Port Proc. Flow/Route Lookup route or state for reserved flows 13 - Jonathan Turner - July 31, 2001 Flow/Route Lookup Distrib. queueing • traffic isolation • protect res. flows Dist. Q. Ctl. Flow Lookup

Improving Datagram Service Per Source Aggregate Queues n. Bandwidth hogging. » single user can take more than fair share of link bandwidth » other users’ packets delayed n. Synchronization of TCP flows. » large queues and large delays sending rate >6. 5 sec. >500 MB queue length 1000 flows at avg. rate of 10 Mb/s 10 Kbits per packet, 100 ms RTT 14 - Jonathan Turner - July 31, 2001 . . . Shared Output Queue n Deficit round-robin service. n Discard policy » longest queue with hysteresis » discard front n Provides traffic isolation. » each queue gets fair share » small delays for “nice” flows n Aggregate queues based on source prefix. » avoid using up queues » limits bandwidth use from single subnet

Super-Scalable Packet Scheduling wheel 1 fast forward bits 00110100 wheel 2 10000010 wheel 3 00101010 output list n n Scalability of Qo. S packet schedulers constrained by need to maintain sorted list of queues. Use approximate radix sorting, with compensation - O(1). » timing wheels with increasing granularity and range » approximate sorting produces inter-packet timing errors » observe errors & compensate when next packet scheduled n n Fast-forward bits used to skip to empty buckets. Scheduler puts no limit on number of queues. 15 - Jonathan Turner - July 31, 2001

n . . . » ensures reserved flows receive assigned bandwidth. . . » allocates unreserved bandwidth fairly to datagram traffic n Periodic broadcast of bandwidth assignments. . Distributed queueing regulates flow of traffic through fabric. Switch Fabric Distributed Queueing . . . » per flow guarantees, without per flow info. broadcast » switch fabric “repackages” data so each port receives only relevant information » update period limited to use <5% of switch bandwidth n adds <100 KB to each input’s buffer space in 1 K port router 16 - Jonathan Turner - July 31, 2001

Prototype Extreme Router Field Programmable Port Ext. Control Processor Smart Port Card Sys. FPGA OPP IPP OPP Pentium APIC IPP North Bridge Switch Fabric OPP 64 MB Cache ATM Switch Core FPX FPX Field Programmable SPC SPC Port Extenders Embedded TI Processors TI Transmisson Interfaces 17 - Jonathan Turner - July 31, 2001 SDRAM FPX 128 MB Input Port Processor SPC VCI TI TI Reprogrammable SPC Application Device VCI OUT TI SRAM 4 MB FPX Network SPC Interface Device TI

Summary Growing reliance on data networks creates higher expectations - reliability, consistent performance. n Design for worst-case - constructive paranoia. n Technology progress making extreme defenses practical, without sacrificing performance. n Extensible, rapidly reconfigurable routers essential. n » reconfigurable hardware, embedded processors Project will develop & evaluate technologies for extreme networking. n Things that haven’t worked. n » PI’s lumbar region » otherwise, too early to say 18 - Jonathan Turner - July 31, 2001