Traffic Engineering for ISP Networks Jennifer Rexford Internet

Traffic Engineering for ISP Networks Jennifer Rexford Internet and Networking Systems AT&T Labs - Research; Florham Park, NJ http: //www. research. att. com/~jrex

Who am I, and Why am I Here? u Who am I? – Princeton EE class of ‘ 91, Ph. D in ‘ 96 from U. Michigan – AT&T Research from 1996, in the IP Network Measurement and Engineering Department – Joining the Princeton CS faculty in February 2005 u Why am I here? – Working closely with operators of AT&T’s IP network – Creating tools for them to engineer the flow of traffic – Applying optimization techniques to real problems

Outline u Background – Internet architecture – Interdomain and intradomain routing – Internet service provider backbone u Traffic engineering – Optimizing routing to prevailing traffic – Local search to select the integer link weights u Inferring the traffic matrix – Working backwards from link loads u Conclusion and ongoing work

Internet Architecture u Divided into Autonomous Systems – Distinct regions of administrative control – Routers and links managed by an institution – Service provider, company, university, … u Hierarchy of Autonomous Systems – Large, tier-1 provider with nationwide backbone – Medium-sized regional provider with smaller backbone – Small network run by a single company or university u Interaction between Autonomous Systems – Internal topology is not shared between ASes – … but, neighboring ASes interact to coordinate routing

Autonomous Systems (ASes) Path: 6, 5, 4, 3, 2, 1 4 3 5 2 1 7 6 Web server Client

Interdomain Routing: Border Gateway Protocol u ASes exchange info about who they can reach – IP prefix: block of destination IP addresses – AS path: sequence of ASes along the path u Policies configured by the AS’s network operator – Path selection: which of the paths to use? – Path export: which neighbors to tell? “I can reach 12. 34. 158. 0/24 via AS 1” “I can reach 12. 34. 158. 0/24” 2 1 data traffic 12. 34. 158. 5 3 data traffic

Internet Service Provider Backbone neighboring providers modem banks, business customers, web/e-mail servers How should traffic be routed through the ISP backbone?

Interior Gateway Protocol (Within an AS) u Routers flood information to learn the topology – Routers determine “next hop” to reach other routers… – By computing shortest paths based on the link weights u Link weights configured by the network operator 2 3 2 1 1 1 3 5 4 3

Heuristics for Setting the Link Weights u Proportional to physical distance – Cross-country links have higher weights than local ones – Minimizes end-to-end propagation delay u Inversely proportional to link capacity – Smaller weights for higher-bandwidth links – Attracts more traffic to links with more capacity u Tuned based on the offered traffic – Network-wide optimization of weights based on traffic – Directly minimizes key metrics like max link utilization

Why Are the Link Weights Static? u Strawman alternative: load-sensitive routing – Link metrics based on traffic load – Flood dynamic metrics as they change – Adapt automatically to changes in offered load u Reasons why this is typically not done – Delay-based routing unsuccessful in the early days – Oscillation as routers adapt to out-of-date information – Most Internet transfers are very short-lived u Research and standards work continues… – … but operators have to do what they can today

Our Approach: Measure, Model, and Control Network-wide “what if” model Offered Topology/ traffic Configuration measure Changes to the network control Operational network

Traffic Engineering in an ISP Backbone u Topology – Connectivity and capacity of routers and links u Traffic matrix – Offered load between points in the network u Link weights – Configurable parameters for Interior Gateway Protocol u Performance objective – Balanced load, low latency, service level agreements … u Question: Given the topology and traffic matrix in an IP network, which link weights should be used?

Key Ingredients of Our Approach u Instrumentation – Topology: monitoring of the routing protocols – Traffic matrix: widely deployed traffic measurement u Network-wide models – Representations of topology and traffic – “What-if” models of shortest-path routing u Network optimization – Efficient algorithms to find good configurations – Operational experience to identify key constraints

Formalizing the Optimization Problem u Input: graph G(R, L) – R is the set of routers – L is the set of unidirectional links – cl is the capacity of link l u Input: traffic matrix – Mi, j is traffic load from router i to j u Output: setting of the link weights – wl is weight on unidirectional link l – Pi, j, l is fraction of traffic from i to j traversing link l

Multiple Shortest Path With Even Splitting 0. 25 0. 5 1. 0 0. 25 0. 5 Values of Pi, j, l

Defining the Objective Function u Computing the link utilization – Link load: ul = Si, j Mi, j Pi, j, l – Utilization: ul/cl u Objective functions – minl(ul/cl) f(x) – minl(S f(ul/cl)) x 1

Complexity of the Optimization Problem u NP-complete optimization problem – No efficient algorithm to find the link weights – Even for the simple convex objective functions u Why can’t we just do multi-commodity flow? – E. g. , solve the multi-commodity flow problem… – … and the link weights pop out as the dual – Because IP routers cannot split arbitrarily over ties u What are the implications? – Have to resort to searching through weight settings

Optimization Based on Local Search u Start with an initial setting of the link weights – E. g. , same integer weight on every link – E. g. , weights inversely proportional to link capacity – E. g. , existing weights in the operational network u Compute the objective function – Compute the all-pairs shortest paths to get Pi, j, l – Apply the traffic matrix Mi, j to get link loads ul – Evaluate the objective function from the ul/cl u Generate a new setting of the link weights repeat

Making the Search Efficient u Avoid repeating the same weight setting – Keep track of past values of the weight setting – … or keep a small signature (e. g. , a hash) of past values – Do not evaluate a weight setting if signatures match u Avoid computing the shortest paths from scratch – Explore weight settings that changes just one weight – Apply fast incremental shortest-path algorithms u Limit the number of unique values of link weights – Do not explore all 216 possible values for each weight u Stop early, before exploring the whole search space

Incorporating Operational Realities u Minimize number of changes to the network – Changing just 1 or 2 link weights is often enough u Tolerate failure of network equipment – Weights settings usually remain good after failure – … or can be fixed by changing one or two weights u Limit dependence on measurement accuracy – Good weights remain good, despite random noise u Limit frequency of changes to the weights – Joint optimization for day and night traffic matrices

Application to AT&T’s Backbone Network u Performance of the optimized weights – Search finds a good solution within a few minutes – Much better than link capacity or physical distance – Competitive with multi-commodity flow solution u How AT&T changes the link weights – Maintenance done every night from midnight to 6 am – Predict effects of removing link(s) from the network – Reoptimize the link weights to avoid congestion – Configure new weights before disabling equipment

Example from My Visit to Our Operations Center u Amtrak repairing/moving part of the train track – Need to move some of the fiber optic cables – Or, heightened risk of the cables being cut – Amtrak notifies us of the time the work will be done u AT&T engineers model the effects – Determine which IP links go over the affected fiber – Pretend the network no longer has these links – Evaluate the new shortest paths and traffic flow – Identify whether link loads will be too high

Example Continued u If load will be too high – Reoptimize the weights on the remaining links – Schedule the time for the new weights to be configured – Roll back to the old weight setting after Amtrak is done u Same process applied to other cases – Assessing the network’s risk to possible failures – Planning for maintenance of existing equipment – Adapting the link weights to installation of new links – Adapting the link weights in response to traffic shifts

Conclusion of This Part of the Talk u IP networks do not adapt on their own – Routers compute shortest paths based on static weights u Service providers need to adapt the weights – Due to failures, congestion, or planned maintenance u Leads to an interesting optimization problems – Optimize link weights based on topology and traffic u Optimization problem in NP-complete – Forces the use of efficient local-search techniques u Results of the local search are useful – Near-optimal solutions that minimize disruptions

Computing the Traffic Matrix Mi, j u Hard to measure the traffic matrix – IP networks transmit data as individual packets – Routers do not keep traffic statistics, except link utilization on (say) a five-minute time scale u Need to infer the traffic matrix Mi, j from – Current topology G(R, L) – Current routing Pi, j, l – Current link load ul – Link capacity cl

Inference: Network Tomography From link counts to the traffic matrix Sources 5 Mbps 3 Mbps 4 Mbps Destinations

Tomography: Formalizing the Problem u Ingress-egress pairs – p is a ingress-egress pair of nodes (i, j) – xp is the (unknown) traffic volume for this pair Mi, j u Routing – Plp is proportion of p’s traffic that traverses l u Links in the network – l is a unidirectional edge – ul is the observed traffic volume on this link u Relationship: u = Px (work backwards to get x)

Tomography: One Observation Not Enough u Linear system of n nodes is underdetermined – Number of links e is around O(n) – Number of ingress-egress pairs c is O(n 2) – Dimension of solution sub-space at least c - e u Multiple observations are needed – k independent observations (over time) – Stochastic model with Poisson iid counts – Maximum likelihood estimation to infer matrix u Doesn’t work all that well in practice…

Approach Used at AT&T: Tomo-gravity u Gravitational assumption – Ingress point a has traffic via – Egress point b has traffic veb – Pair (a, b) has traffic proportional to via * veb 9 6 3 20 10

Approach Used at AT&T: Tomo-gravity u Problem with gravity model – Gravity model ignores the load on the inside links – Gravity assumption isn’t always 100% correct – Resulting traffic matrix might not satisfy the link loads u Combining the two techniques – Gravity: find a traffic matrix using the gravity model – Tomography: find the family of traffic matrices consistent with all link load statistics – Tomo-gravity: find the tomography solution that is closest to the output of the gravity model u Works extremely well (and fast) in practice

Conclusions u Managing IP networks is challenging – Routers don’t adapt on their own to congestion – Routers don’t reveal much information about traffic u Measurement provides a network-wide view – Topology – Traffic matrix u Optimization enables the network to adapt – Inferring the traffic matrix from the link loads – Optimizing the link weights based on the traffic matrix