Traffic Engineering for ISP Networks Jennifer Rexford Computer

Traffic Engineering for ISP Networks Jennifer Rexford Computer Science Department Princeton University http: //www. cs. princeton. edu/~jrex

Outline u Overview of Internet routing – IP addressing and forwarding – Interdomain and intradomain routing u Optimization: Tuning routing to the traffic – Optimizing routing given a topology and traffic matrix – Local search to select the integer link weights u Tomography: Inferring the traffic matrix – Estimating traffic matrix from routing and link loads u Conclusion and ongoing work

IP Service Model: Best-Effort Packet Delivery u Packet switching – Send data in packets – Header with source and destination address u Best-effort delivery – Packets may be lost – Packets may be corrupted – Packets may be delivered out of order source destination IP network

Packet Delivery Based on Destination IP Address u 32 -bit number in dotted-quad notation (12. 34. 158. 5) u Divided into network & host portions (left and right) u 12. 34. 158. 0/24 12 is a 24 -bit prefix with 28 addresses 34 158 5 00001100 0010 10011110 00000101 Network (24 bits) Host (8 bits)

Longest-Prefix Match Forwarding u Forwarding tables in IP routers – Maps each IP prefix to next-hop link(s) u Destination-based forwarding – Packet has a destination address – Router identifies longest-matching prefix forwarding table destination 12. 34. 158. 5 4. 0. 0. 0/8 4. 83. 128. 0/17 12. 0. 0. 0/8 12. 34. 158. 0/24 126. 255. 103. 0/24 outgoing link Serial 0/0. 1

Where do Forwarding Tables Come From? u Routers have forwarding tables – Map prefix to outgoing link(s) u Entries can be statically configured – E. g. , “map 12. 34. 158. 0/24 to Serial 0/0. 1” u But, this doesn’t adapt – To failures – To new equipment – To the need to balance load u That is where routing protocols come in…

Two-Tiered Internet Routing Architecture u Goal: distributed management of resources – Internetworking of multiple networks – Networks under separate administrative control u Solution: two-tiered routing architecture – Intradomain: inside a region of control » Okay for routers to share topology information » Routers configured to achieve a common goal – Interdomain: between regions of control » Not okay to share complete information » Networks may have different/conflicting goals

Autonomous Systems (ASes) u Autonomous Systems – Distinct regions of administrative control – Routers and links managed by an institution – Service provider, company, university, … u AS hierarchy – Tier-1 provider with national or global backbone – Regional provider with smaller backbone – Campus or corporate network u Interaction between ASes – Internal topology is not shared between ASes – … but, neighboring ASes interact to coordinate routing

AS Numbers (ASNs) Currently around 25, 000 in use. • • • Level 3: 1 MIT: 3 Harvard: 11 Yale: 29 Princeton: 88 AT&T: 7018, 6341, 5074, … UUNET: 701, 702, 284, 12199, … Sprint: 1239, 1240, 6211, 6242, … … ASNs represent units of routing policy

Traffic Traverses Multiple ASes Path: 6, 5, 4, 3, 2, 1 4 3 5 2 1 7 6 Web server Client

Interdomain Routing: Border Gateway Protocol u ASes exchange info about who they can reach – IP prefix: block of destination IP addresses – AS path: sequence of ASes along the path u Policies configured by the AS’s network operator – Path selection: which of the paths to use? – Path export: which neighbors to tell? “I can reach 12. 34. 158. 0/24 via AS 1” “I can reach 12. 34. 158. 0/24” 2 1 data traffic 12. 34. 158. 5 3 data traffic

Interior Gateway Protocol (Within an AS) u Routers flood information to learn the topology – Routers determine “next hop” to reach other routers… – By computing shortest paths based on the link weights u Link weights configured by the network operator 2 3 2 Serial 0/0. 1 1 1 4 1 3 5 3 12. 34. 158. 0/24

Constructing the Forwarding Table u Two routing protocols – BGP: learn the external route at some border router – IGP: learn outgoing link on path to other router u Router joins the data – Prefix 12. 34. 158. 0/24 reached through red router – Red router reached via link Serial 0/0. 1 – Forwarding entry: 12. 34. 158. 0/24 Serial 0/0. 1 u Router forwards packets – Lookup destination 12. 34. 158. 5 in table – Forward packet out link Serial 0/0. 1

Optimization: Tuning Routing to the Traffic

Link Weights Control the Flow of Traffic u Routers compute paths – Shortest paths as sum of link weights u Operators set the link weights – To control where the traffic goes 2 3 2 1 1 31 3 5 4 3

Heuristics for Setting the Link Weights u Proportional to physical distance – Cross-country links have higher weights than local ones – Minimizes end-to-end propagation delay u Inversely proportional to link capacity – Smaller weights for higher-bandwidth links – Attracts more traffic to links with more capacity u Tuned based on the offered traffic – Network-wide optimization of weights based on traffic – Directly minimizes key metrics like max link utilization

Why Are the Link Weights Static? u Strawman alternative: load-sensitive routing – Link metrics based on traffic load – Flood dynamic metrics as they change – Adapt automatically to changes in offered load u Reasons why this is typically not done – Delay-based routing unsuccessful in the early days – Oscillation as routers adapt to out-of-date information – Most Internet transfers are very short-lived u Research and standards work continues… – … but operators have to do what they can today

Big Picture: Measure, Model, and Control Network-wide “what if” model Offered Topology/ traffic Configuration measure Changes to the network control Operational network

Traffic Engineering in an ISP Backbone u Topology – Connectivity and capacity of routers and links u Traffic matrix – Offered load between points in the network u Link weights – Configurable parameters for Interior Gateway Protocol u Performance objective – Balanced load, low latency, service level agreements … u Question: Given the topology and traffic matrix in an IP network, which link weights should be used?

Key Ingredients of Our Approach u Measurement – Topology: monitoring of the routing protocols – Traffic matrix: widely deployed traffic measurement u Network-wide models – Representations of topology and traffic – “What-if” models of shortest-path routing u Network optimization – Efficient algorithms to find good configurations – Operational experience to identify key constraints

Formalizing the Optimization Problem u Input: graph G(R, L) – R is the set of routers – L is the set of unidirectional links – cl is the capacity of link l u Input: traffic matrix – Mi, j is traffic load from router i to j u Output: setting of the link weights – wl is weight on unidirectional link l – Pi, j, l is fraction of traffic from i to j traversing link l

Multiple Shortest Paths With Even Splitting 0. 25 0. 5 1. 0 0. 25 0. 5 Values of Pi, j, l

Defining the Objective Function u Computing the link utilization – Link load: ul = Si, j Mi, j Pi, j, l – Utilization: ul/cl u Objective functions – min(maxl(ul/cl)) f(x) – min(Sl f(ul/cl)) x 1

Complexity of the Optimization Problem u NP-hard optimization problem – No efficient algorithm to find the link weights – Even for the simple convex objective functions u Why can’t we just do multi-commodity flow? – E. g. , solve the multi-commodity flow problem… – … and the link weights pop out as the dual – Because IP routers cannot split arbitrarily over ties u What are the implications? – Have to resort to searching through weight settings

Optimization Based on Local Search u Start with an initial setting of the link weights – E. g. , same integer weight on every link – E. g. , weights inversely proportional to link capacity – E. g. , existing weights in the operational network u Compute the objective function – Compute the all-pairs shortest paths to get Pi, j, l – Apply the traffic matrix Mi, j to get link loads ul – Evaluate the objective function from the ul/cl u Generate a new setting of the link weights repeat

Making the Search Efficient u Avoid repeating the same weight setting – Keep track of past values of the weight setting – … or keep a small signature (e. g. , a hash) of past values – Do not evaluate a weight setting if signatures match u Avoid computing the shortest paths from scratch – Explore weight settings that changes just one weight – Apply fast incremental shortest-path algorithms u Limit the number of unique values of link weights – Do not explore all 216 possible values for each weight u Stop early, before exploring the whole search space

Incorporating Operational Realities u Minimize number of changes to the network – Changing just 1 or 2 link weights is often enough u Tolerate failure of network equipment – Weights settings usually remain good after failure – … or can be fixed by changing one or two weights u Limit dependence on measurement accuracy – Good weights remain good, despite random noise u Limit frequency of changes to the weights – Joint optimization for day and night traffic matrices

Application to AT&T’s Backbone Network u Performance of the optimized weights – Search finds a good solution within a few minutes – Much better than link capacity or physical distance – Competitive with multi-commodity flow solution u How AT&T changes the link weights – Maintenance done every night from midnight to 6 am – Predict effects of removing link(s) from the network – Reoptimize the link weights to avoid congestion – Configure new weights before disabling equipment

Example from My Visit to AT&T’s Operations Center u Amtrak repairing/moving part of the train track – Need to move some of the fiber optic cables – Or, heightened risk of the cables being cut – Amtrak notifies us of the time the work will be done u AT&T engineers model the effects – Determine which IP links go over the affected fiber – Pretend the network no longer has these links – Evaluate the new shortest paths and traffic flow – Identify whether link loads will be too high

Example Continued u If load will be too high – Reoptimize the weights on the remaining links – Schedule the time for the new weights to be configured – Roll back to the old weight setting after Amtrak is done u Same process applied to other cases – Assessing the network’s risk to possible failures – Planning for maintenance of existing equipment – Adapting the link weights to installation of new links – Adapting the link weights in response to traffic shifts

Conclusions on Traffic Engineering u IP networks do not adapt on their own – Routers compute shortest paths based on static weights u Service providers need to adapt the weights – Due to failures, congestion, or planned maintenance u Leads to an interesting optimization problems – Optimize link weights based on topology and traffic u Optimization problem is computationally difficult – Forces the use of efficient local-search techniques u Results of the local search are good – Near-optimal solutions that minimize disruptions

Ongoing Work u Robust link-weight assignments – Link/node failures – Range of traffic matrices u More complex routing models – Hot-potato routing – BGP routing policies u Interaction between ASes – Inter-AS negotiation for joint optimization – Grappling with scalability and trust issues

Tomography: Inferring the Traffic Matrix

Computing the Traffic Matrix Mi, j u Hard to measure the traffic matrix – IP networks transmit data as individual packets – Routers do not keep traffic statistics, except link utilization on (say) a five-minute time scale u Need to infer the traffic matrix Mi, j from – Current topology G(R, L) – Current routing Pi, j, l – Current link load ul – Link capacity cl

Inference: Network Tomography From link counts to the traffic matrix Sources 5 Mbps 3 Mbps 4 Mbps Destinations

Tomography: Formalizing the Problem u Ingress-egress pairs – p is a ingress-egress pair of nodes (i, j) – xp is the (unknown) traffic volume for this pair Mi, j u Routing – Plp is proportion of p’s traffic that traverses l u Links in the network – l is a unidirectional edge – ul is the observed traffic volume on this link u Relationship: u = Px (work backwards to get x)

Tomography: One Observation Not Enough u Linear system of n nodes is underdetermined – Number of links e is around O(n) – Number of ingress-egress pairs c is O(n 2) – Dimension of solution sub-space at least c - e u Multiple observations are needed – k independent observations (over time) – Stochastic model with Poisson iid counts – Maximum likelihood estimation to infer matrix u Doesn’t work all that well in practice…

Approach Used at AT&T: Tomo-gravity u Gravitational assumption – Ingress point a has traffic via – Egress point b has traffic veb – Pair (a, b) has traffic proportional to via * veb 9 6 3 20 10

Approach Used at AT&T: Tomo-gravity u Problem with gravity model – Gravity model ignores the load on the inside links – Gravity assumption isn’t always 100% correct – Resulting traffic matrix might not satisfy the link loads u Combining the two techniques – Gravity: find a traffic matrix using the gravity model – Tomography: find the family of traffic matrices consistent with all link load statistics – Tomo-gravity: find the tomography solution that is closest to the output of the gravity model u Works extremely well (and fast) in practice

Conclusions u Managing IP networks is challenging – Routers don’t adapt on their own to congestion – Routers don’t reveal much information about traffic u Measurement provides a network-wide view – Topology – Traffic matrix u Optimization enables the network to adapt – Inferring the traffic matrix from the link loads – Optimizing the link weights based on the traffic matrix

New Research Direction: Design for Manage-ability u Two main parts of network management – Control: optimization – Measurement: tomography u Two research approaches – Bottom up: do the best with what you have – Top down: design systems that are easier to manage u Design for manage-ability – “If you are both the professor and the student, you create exam questions that are easy to answer. ” – Mung Chiang