Dynamics of HotPotato Routing in IP Networks Jennifer

Dynamics of Hot-Potato Routing in IP Networks Jennifer Rexford AT&T Labs—Research http: //www. research. att. com/~jrex Joint work with Renata Teixeira (UCSD), Aman Shaikh (AT&T), and Timothy Griffin (Intel)

Outline w Internet routing n n Interdomain and intradomain routing Coupling due to hot-potato routing w Measuring hot-potato routing n n Measuring the two routing protocols Correlating the two data streams w Performance evaluation n Characterization on AT&T’s network w Implications on operational practices w Conclusions and future directions 2

Autonomous Systems Multiple links Middle of path 4 3 5 2 7 1 6 Web server Client AS path: 6, 5, 4, 3, 2, 1 3

Interdomain Routing (BGP) w Border Gateway Protocol (BGP) n n IP prefix: block of destination IP addresses AS path: sequence of ASes along the path w Policy configuration by the operator n n Path selection: which of the paths to use? Path export: which neighbors to tell? “I can reach 12. 34. 158. 0/24 via AS 1” “I can reach 12. 34. 158. 0/24” 1 12. 34. 158. 5 2 3 4

Intradomain Routing (IGP) w Interior Gateway Protocol (OSPF and IS-IS) n n n Shortest path routing based on link weights Routers flood link-state information to each other Routers compute “next hop” to reach other routers w Link weights configured by the operator n n Simple heuristics: link capacity or physical distance Traffic engineering: tuning link weights to the traffic 2 3 2 Path cost: 2+1+5 1 1 4 3 1 5 3 5

Two-Level Internet Routing inter-domain routing (BGP) AS 3 AS 2 w Hierarchical routing n Intra-domain l n Metric based Inter-domain l Reachability and policy w Design principles AS 1 intra-domain routing (IGP) n n n Autonomous system (AS) = network with unified administrative routing policy (ex. AT&T, Sprint, UCSD) Scalability Isolation Simplicity of reasoning 6

Coupling: Hot-Potato Routing Z X dst 10 ISP network Y Routes thousands packet to dst of destinations switch exit point!!! 11 9 failure planned maintenance traffic engineering Consequences: w. Routers CPU overload Hot-potato routing = ISPs policy of routing to w. Transient forwarding instability closest exit point when there w. Traffic is more than one route shift to destination w. Inter-domain routing changes 7

BGP Decision Process “Equally good” w Ignore if exit point unreachable w Highest local preference w Lowest AS path length w Lowest origin type w Lowest MED (with same next hop AS) w Lowest IGP cost to next hop w Lowest router ID of BGP speaker 8

Hot-Potato Routing Model Z 10 X 99 W dst 8 Y w Cost vector for Z: c. X=10, c. W=8, and c. Y=9 w Egress set for dst: {X, Y} w Best route for Z: through Y, which is closest Hot-potato change: change in cost vector causes change in best route 9

The Big Picture Cost vector changes Interdomain changes Hot-potato routing changes (interdomain changes caused by intradomain changes) 10

Why Care about Hot Potatoes? w Understanding of Internet routing n n Frequency of hot-potato routing changes Influence on end-to-end performance w Operational practices n n n Knowing when hot-potato changes happen Avoiding unnecessary hot-potato changes Analyzing externally-caused BGP updates w Distributed root cause analysis n n Each AS can tell what BGP updates it caused Someone should know why each change happened 11

Our Approach w Measure both protocols n BGP and OSPF monitors Z X AT&T backbone OSPF messages BGP updates M Y w Correlate the two streams n Match BGP updates with OSPF events w Analyze the interaction 12

Heuristic for Matching Stream of OSPF messages Transform stream of OSPF messages into routing changes refresh link failure weight change chg cost del Match BGP updates with OSPF events that happen close in time Classify BGP updates by possible OSPF causes time Stream of BGP updates 13

Computing Cost Vectors w Transform OSPF messages into path cost changes from a router’s perspective OSPF routing changes: X 11 1 2 2 10 10 2 1 1 Z X X 55 Y Y 4 7 CHG Y, 7 DEL X ADD X, 5 M LSA weight change, 10 10 LSA deletechange, Y 14

Classifying BGP Updates w Cannot have been caused by cost change n n n Destination just became (un)available in BGP New BGP route through same egress point New route better/worse than old (e. g. , AS path len) w Can have been caused by cost change n New route is equally good as old route (perhaps X got closer, or Y got further away) M Z X dst Y 15

The Role of Time w IGP link-state advertisements n n Multiple LSAs from a single physical event Group into single cost vector change w BGP update messages n n n 70 sec Multiple BGP updates during convergence Group into single BGP routing change w Matching IGP to BGP n 10 sec 180 sec Avoid matching unrelated IGP and BGP changes Match related changes that are close in time Characterize the measurement data to determine the right windows 16

Summary Results (June 2003) w High variability in % of BGP updates location min max days > 10% close to peers 0% 3. 76% 0 between peers 0% 25. 87% 5 w One LSA can have a big impact location no impact prefixes impacted close to peers 97. 53% less than 1% between peers 97. 17% 55% 17

Delay for BGP Routing Change w Router vendor scan timer n n BGP table scan every 60 seconds OSPF changes arrive uniformly in interval w Internal BGP hierarchy (route reflectors) n n Wait for another router to change best route Introduces a wait for a second BGP scan w Transmitting many BGP messages n Latency for transferring the data We have seen delays of up to 180 seconds! 18

Delay for First BGP Change Fraction of Hot-Potato Changes Two BGP scans Cisco BGP scan timer Routers in backbone (June) Routers in backbone (September) Router in lab time BGP update – time LSA (seconds) 19

Cumulative Number of Hot-Potato Changes Transferring Multiple Prefixes 81 seconds delay time BGP update – time LSA (seconds) 20

BGP Updates Over Prefixes Contrast with non-OSPF triggered BGP updates Cumulative %BGP updates prefixes with only one exit point OSPF-triggered BGP updates affects ~50% of prefixes uniformly Non-OSPF triggered All OSPF-triggered % prefixes 21

Operational Implications w Forwarding plane convergence n Accuracy of active measurements w Router proximity to exit points n Likelihood of hot-potato routing changes w Cost in/out of links during maintenance n Avoid triggering BGP routing changes 22

Forwarding Convergence R 1’s scan process Packets to dst may can take up to be caught Scan processin a loop using to reach dst 60 seconds to R run for in 60 Rseconds! 2 startsruns 1 2 R 1 10 R 2 111 10 100 dst 23

Measurement Accuracy w Measurements of customer experience n Probe machines have just one exit point! loop to reach dst W 1 R 1 10 R 2 111 100 W 2 dst 24

Avoid Equal-distance Exits Z X 10 Z X 1 10 1000 Y dst Small changes will make Z switch exit points to dst More robust to intra-domain routing changes 25

Careful Cost in/out Links X 5 100 5 10 4 Z 10 10 dst Y Traffic is more predictable Faster convergence Less impact on neighbors 26

Ongoing Work w Black-box testing of the routers n n Scan timer and its effects (forwarding loops) Vendor interactions (with Cisco) w Impact of the IGP-triggered BGP updates n n n Changes in the flow of traffic Forwarding loops during convergence Externally visible BGP routing changes w Improving isolation (cooling those potatoes!) n n n Operational practices: preventing interactions Protocol design: weaken the IGP/BGP coupling Network design: internal topology/architecture 27

Thanks! w Any questions? 28

i. BGP Route Reflectors Announcement X X 9 Y dst 8 11 10 Z dst X, 19 Y, 21 18 dst W, 20 20 W Scalability trade-off: Less BGP state vs. Number of BGP updates from Z and longer convergence delay 29

Exporting Routing Instability Announcement Z X dst Y No change => no announcement Z X dst Y 30