BGP Anomaly Detection in an ISP Jian Wu

BGP Anomaly Detection in an ISP Jian Wu (U. Michigan) Z. Morley Mao (U. Michigan) Jennifer Rexford (Princeton) Jia Wang (AT&T Labs) http: //www. cs. princeton. edu/~jrex/papers/nsdi 05 -jian. pdf 1

Goal o Identify important anomalies n n n Lost reachability Persistent flapping Large traffic shifts Contributions: • Build a tool to identify a small number of important routing disruptions from a large volume of raw BGP updates in real time. • Use the tool to characterize routing disruptions in an operational network 2

Capturing Routing Changes tes BR BR C i. BGP Bes t ro utes GP P G e. B BR BR C U s st Be te rou BR BR C P e. BG P i. BG BGP CPE Monitor i. BGP i. B es t pda i. BGP Upd a e. BGP Large operational network (8/16/2004 – 10/10 -2004) BR BR C i. BG P BR BR C e. BG P P e. BG 3

Challenges o Large volume of BGP updates n n o Millions daily, very bursty Too much for an operator to manage Different than root-cause analysis n n n Identify changes and their effects Focus on actionable events Diagnose causes only in/near the AS 4

System Architecture BGP (106) BR E Updates Events (105) BR E BGP Update Grouping Persistent Flapping Prefixes (101) “Typed” Events Event Classification Clusters Event Correlation Frequent Flapping Prefixes (101) Large Disruptions (101) (103) Traffic Impact Prediction Netflow Data BR E 5

Grouping BGP Update into Events Challenge: A single routing change n n leads to multiple update messages affects routing decisions at multiple routers BR E BGP Updates BR E BGP Update Grouping Persistent Flapping Prefixes Events Solution: • Group all updates for a prefix with interarrival < 70 seconds • Flag prefixes with changes lasting > 10 minutes. 6

Grouping Thresholds o o Based on data analysis and our understanding of BGP Event timeout: 70 seconds n n o 2 * MRAI timer + 10 seconds 98% inter-arrival time < 70 seconds Convergence timeout: 10 minutes n n BGP usually converges within minutes 99. 9% events < 10 minutes 7

Persistent Flapping Prefixes Surprising finding: 15. 2% of updates were caused by persistent flapping prefixes, even though flap damping was enabled! o Causes of persistent flapping n Conservative damping parameters (78. 6%) n Protocol oscillations due to MED (18. 3%) n Unstable interface or BGP session (3. 0%) 8

Example: Unstable e. BGP Session ISP AE DE Peer BE CE p Customer o o Flap damping parameters are session-based Damping not implemented for i. BGP sessions 9

Event Classification Challenge: Major concerns in network management n Changes in reachability n Heavy load of routing messages on the routers n Change of flow of traffic through the network Events Event Classification “Typed” Events Solution: classify events by severity of their impacts 10

Event Category – “No Disruption” p AS 2 AS 1 DE No Traffic Shift EE AE BE “No Disruption”: each. ISP of the border routers has no traffic shift. (50. 3%) CE 11

Event Category – “Internal Disruption” p AS 2 AS 1 DE EE AE BE “Internal Disruption”: ISP all of the traffic shifts are internal traffic shift. (15. 6%) CE Internal Traffic Shift 12

Event Category – “Single External Disruption” p AS 2 AS 1 DE external Traffic Shift EE AE BE “Single External Disruption”: only one of the ISP traffic shifts is external traffic shift. (20. 7%) CE 13

Statistics on Event Classification Events Updates No Disruption 50. 3% 48. 6% Internal Disruption 15. 6% 3. 4% Single External Disruption 20. 7% 7. 9% 7. 4% 18. 2% Multiple External Disruption Loss/Gain of categories have 6. 0% o First 3 significant 21. 9% variations from Reachability day to day o Updates per event depends on the type of events and the number of affected routers 14

Event Correlation Challenge: A single routing change n affects multiple destination prefixes “Typed” Events Event Correlation Clusters Solution: group events of same type that occur close in time 15

EBGP Session Reset o o Caused most “single external disruption” events Check if the number of prefixes using that session as the best route changes dramatically Number of prefixes session recovery session failure time o Validation with Syslog router report (95%) 16

Hot-Potato Changes o Hot-Potato Changes P AE 11 9 BE ISP 10 “Hot-potato routing” = route to closest egress point CE o o Caused “internal disruption” events Validation with OSPF measurement (95%) [Teixeira et al – SIGMETRICS’ 04] 17

Traffic Impact Prediction Challenge: Routing changes have different impacts on the network which depends on the popularity of the destinations Traffic Impact Prediction Clusters Large Disruptions Netflow Data E BR Solution: weigh each cluster by traffic volume 18

Traffic Impact Prediction o Traffic weight n n o Per-prefix measurement from Netflow 10% prefixes accounts for 90% of traffic Traffic weight of a cluster n n n Sum of “traffic weight” of the prefixes A few clusters have large traffic weight Mostly session resets & hot-potato changes 19

Performance Evaluation o Memory n n o Static memory: “current routes”, 600 MB Dynamic memory: “clusters”, 300 MB Speed n n n 99% of intervals of 1 second of updates can be process within 1 second Occasional execution lag Every interval of 70 seconds of updates can be processed within 70 seconds Measurements were based on 900 MHz CPU 20

Conclusion o BGP anomaly detection n o Fast, online fashion Operator concerns (reachability, flapping, traffic) Significant information reduction Uncovered important network behaviors n n n Persistent flapping prefixes Hot-potato changes Session resets and interface failures 21

Detecting Peering Violations o Consistent export requirement n n o Detecting violations n n o Peer should advertise prefixes at all peering points, with the same AS path length Allows the AS to do hot-potato routing Using i. BGP feeds from the border routers Some inference tricks to identify inconsistencies Results of the study n http: //www. nanog. org/mtg-0410/feamster. html n http: //www. cs. princeton. edu/~jrex/papers/imc 04. pdf 22