BGP Introduction and Issues Michalis Faloutsos with the

BGP: Introduction and Issues Michalis Faloutsos (with the help of various contributions of slides) Advanced Networks 1

What Is BGP? Border Gateway Protocol BGP-4 The de-facto interdomain routing protocol BGP includes specifications: • Which information gets advertised and how BGP includes a routing protocol: • Establishes and uses a routing table Internal Gateway Protocol (I-BGP in the book) Advanced Networks 2

Why Is There Such Fuss about BGP? BGP dictates routing at the AS level • Absence of understanding: poor performance BGP is complicated • Designed to be flexible • Involves multiple fields Understanding BGP behavior is not intuitive • Implementation and business policies The routing of the Internet relies on BGP Advanced Networks 3

Some Open Questions How well does BGP work now? How can I manage a BGP network? How secure and robust is BGP? • Cyber-terrorism How would we re-design BGP now? How well will BGP scale for our future needs? Advanced Networks 4

Roadmap Introduction to BGP Highlights of BGP issues Goal: instigate interest in BGP Advanced Networks 5

Some Basic Numbers • 43, 000 Autonomous Systems approx. • Corporate Networks • ISP Internal Networks • National Service Providers • Identified by ASN a 16 bit value • Assigned by IANA • Superlinear growth (Huston, Siganos et al. ) Advanced Networks 6

How A BGP graph Looks Like AS 2 AS 5 AS 4 AS 3 Each AS has designated BGP routers of an AS communicate internally with another protocol (IGP) AS 1 Advanced Networks 7

IP Addresses and Prefixes IP addresses have 32 bits: 4 octets of bits (IPv 4) A prefix is a group of IP addresses 128. 32. 101. 5 is an IP address (32 bits) 128. 32. 0. 0/16 is a prefix of the 16 first bits: • 128. 32. 0. 0 – 128. 32. 255 (2^16 addresses) 128. 32. 4. 0/24 is a prefix of the 24 first bits longer Advanced Networks 8

Routing is Based on Prefixes A BGP Routing table has prefixes for entries For a IP address of a packet, find longest match Example: packet IP 128. 32. 101. 1 128. 1. 1. 4 matches the first 8 bits – no match! 128. 32. 0. 0/16 match for 16 bits 128. 32. 101. 0/24 is a longer match Advanced Networks 9

Prefix Matching in More Detail For a IP address of a packet, find longest match Example: Compare • • packet IP 128. 32. 101. 1 With 128. 32. 0. 0/16 IP : 01000000. 01100101. 00000001 Mask : 11111. 00000000 AND : 0100000000. 0000 Prefix : 0100000000. 0000 Equal? Yes Advanced Networks 10

Advertising Routing Information Each AS advertises what it can reach from each BGP router Policies I: filter what you advertise Policies II: filter from what you hear advertised Build up a BGP routing table • Remember which prefix you hear from which link Advanced Networks 11

What Does a Routing Table Look Like? Prefix Origin AS Path 128. 32. 0. 0/16 123 14 56 123 34 101 203 123 15 50 15 15 128. 32. 101. 0/24 Origin AS “owns” the address Routing tables can have peculiarities Advanced Networks 12

Route Advertising Distance Vector style protocol Hear advertisements: IP prefix, AS-path Filter if desired (i. e. ignore) Append yourself: IP prefix, my. AS+AS-path Forward to appropriate ASs Advanced Networks 13

Basic AS relationships Customer – Provider • Customer pays Provider for service • The Customer is always right Peer to Peer: mutual cooperation • Ex. MCI and AT&T Sibling-Sibling • Ex. AT&T research and AT&T wireless Advanced Networks 14

The Internet as a Directed Graph Every edge is bidirectional reg. traffic! Business relationships are represented Provider Peer Advanced Networks Customer Peer 15

The Initial Idea Data flows between customers-providers Top level providers are peers • They exchange information to ensure connectivity What can possibly go wrong? Advanced Networks 16

And then came the rain… Thousands of ASs Complicated relationships Multiple providers for one AS, and IP prefix!! • Multihoming Traffic engineering • I want to use multiple paths and load balance Advanced Networks 17

AS Relationships 200 10 11 1 2 12 3 Provider Peer Customer Peer 13 4 Customer – Provider: customer pays and is always right Peer to Peer: Exchange traffic only between their customers Sibling-Sibling: Exchange traffic at will Advanced Networks 18

The Rules of BGP Routing Transit traffic: traffic that does not go to my customers (or their customers) A provider carries any traffic to, from customer Peers exchange traffic only if between their customers Advanced Networks 19

How BGP Policy Restricts Routing Provider 100 200 Peer Customer Peer 10 11 1 2 12 13 3 4 Path Properties: Routing rules: • Provider accept everything • Peer only if it is for its customers • Up then down • No up-down-up, at most 1 peer-peer steps Advanced Networks 20

What paths are allowed? Provider 100 200 Peer Customer Peer 10 11 1 2 12 13 3 4 Advanced Networks 21

Implementing BGP Rules What do you do with an advertisement: Through customer link • Advertise to all (customers, peers, providers) Through sibling link • Advertise to all Through provider link • Advertise to customer only (and possibly siblings) Through peer link • Advertise to customer only (and possibly siblings) Advanced Networks 22

How Policies Affect Routing Customer 1 ISP 1 A Provider will get rid of traffic as soon as possible, But a Provider will carry the traffic for its customer Did anyone say traffic is asymmetric? ISP 2 Customer 2 Advanced Networks 23

BGP Path-Length Asymmetry Consider number of AS traversed by a path Asymmetry: 46% of pairs differ by at least one AS hop [Siganos 01] Advanced Networks 24

Determining The Logical Graph The business relationships are critical How can I find the relationships? 1. Infer relationships from routing tables 2. IRR database: manually maintained – error prone Advanced Networks 25

Two Inference Algorithms Inference algorithm [Gao 00] • Using 1 routing table • Exploit the up-down path property in a routing path, assume highest degree node as peak Inference using multiple tables [Subramanian 02] • Use multiple points of observation to improve results Advanced Networks 26

Things Become Interesting: Traffic Engineering AS 2 LP 100 LP 80 slow 208. 1. 1. 0/24 How can I pick a route? Local Preference: path attribute AS 2 wants to prefer fast thick link Advertisement from right router of AS 2 has higher Local Preference Any BGP router in AS 2 will prefer the thick link AS 1 Advanced Networks 27

Load Balancing - Appetizer I want to share traffic between my two providers How can I do this? Advanced Networks 28

Load Balancing: Long Prefix Match Takes All! 138. 39/16 ISP 3 138. 39. 1/24 ISP 2 138. 39. 1/24 ISP 1 138. 39/16 138. 39. 1/24 Customer 138. 39. 1/24 Advanced Networks 29

So How Can I Balance the Load? Ask my provider to not aggregate my prefix • Will this work? Split my prefix in two • 138. 39. 1. 0/24 • A: 138. 39. 1. 31/28 • B: 138. 39. 1. 32/28 Advertise only one part to ISP 2 traffic destined for prefixes in A ISP 1 traffic destined for prefixes in B Advanced Networks 30

Summary Up to Now BGP-4 is the de facto protocol for interdomain routing BGP was developed to achieve: • Flexible policy implementation • Scalability via route aggregation given CIDR There are many open issues • BGP is a hot research topic Advanced Networks 31

The Growth of BGP Table Advanced Networks 32

The BGP Growth: The Truth Growth flattened out Why? • Better management • Dot-com crash? Advanced Networks 33

Routing Table Variation 100 k Larger ASes have significantly larger tables Advanced Networks 34

Update Activity Per Prefix Measure rate of announcements + withdrawals + path updates Compare relative update rate per prefix length to the relative number of prefixes of that length >1 implies higher than average update rate (less stable) • <1 implies lower than average update rate(more stable) • Advanced Networks 35

Measured Update Rate Bursty! Advanced Networks 36

BGP robustness Measuring the BGP updates Advanced Networks 37

Analyzing Messages By Content Aggregated per 30 seconds Advanced Networks 38

Initial Observations Updates show daily and weekly periodicity There is no evidence of BGP disturbance: • The Baltimore tunnel train 18 July that destroyed Internet lines • Sept 11 attack There are some spikes at: • • 19 July 18 -22 September Advanced Networks 39

BGP Updates Correlations Advanced Networks 40

BGP Under Attack Advanced Networks 41

Router CPU Activity Correlates… Advanced Networks 42

The Attack of The Worm Advanced Networks 43

Classification of Instabilities Advanced Networks 44

Isolating Instability: 1 unstable peer Advanced Networks 45

Detecting abnormal BGP activity Advanced Networks 46

The Worm Activity Advanced Networks 47

The Worm Correlates Again… Advanced Networks 48

Advanced Networks 49

Advanced Networks 50

Advanced Networks 51

Advanced Networks 53

Advanced Networks 54

Summary of BGP Instability Globally correlated BGP instability is not uncommon Some causes are well understood (misconfiguration, bad path announcements) Some others are less well understood, and more worrisome: • worms Advanced Networks 55

BGP: Up Close and Personal Establishing a connection Messages Path Attributes Advanced Networks 56

Establishing A BGP Session BGP uses TCP connections • For reliability A BGP session is between two routers • Typically directly connected (Ethernet, FDDI) Routers establish a BGP session • • • Authentication and set-up Update and withdrawals If disconnected, all paths are invalidated Advanced Networks 57

Messages First open TCP connection • Identification and authentication BGP messages • • OPEN: set-up, negotiate timer for keep-alive UPDATE: routing changes NOTIFICATION: termination, and error messages KEEPALIVE: confirm that connection is active Advanced Networks 58

UPDATE Message Advertise reachability information Withdraw paths to prefixes Update information prefixes Introduce new prefixes Modify important path attributes for new prefixes and the related paths Advanced Networks 59

Standard Path Attributes Attribute: Type, Length, Value Origin: where did I hear this from • BGP (external) or IGP (internal) AS Path: sequence of ASs Flexible handling of loops • Recovering from disconnected ASs! Next Hop: set explicitly who the next router should be (possibly a non BGP speaker) Advanced Networks 60

Multi-Exit Discriminator (MED) MED 10 AS 2 AS 1 MED 50 • Indication to external peers of the preferred path into the AS • Lowest Med Preferred Advanced Networks 61

Note for MED Defines preference for incoming traffic One AS sets the values Another AS interprets and uses them Thus: • • We need cooperative ASes Only between two ASes (1 hop scope) MED is meaningless in the next hop MED can be used only if both routes are advertised from the same AS Advanced Networks 62

Local Preference AS 2 AS 1 L 1 Choosing paths internally • Defining outgoing traffic Within an AS: • Set Local Preference to pick the path you want to send data to L 2 The higher Local AS 3 Preference is Here MED can not be used since we preferred have different AS Advanced Networks 63

Aggregation of updates AS want to aggregate as much as possible • Reduce routing state • Reduce information that needs to be exchanged Main idea: send one update instead of two • All other attributes are the same (path, preferences etc) • The prefixes have to be subsets, or adjacent subset adjacent Advanced Networks 64

Atomic Aggregate Sometimes we aggregate paths that are different (ie different AS sequence) Atomic aggregate shows that some of the destinations in this prefix are not necessarily following the same path This does not allow other routers to deaggregate the path, thus creating entries that should not appear Advanced Networks 65

Atomic Aggregate 138. 39/16 : 14 AS 3 138. 39/16 : 314 AS 1 138. 39/16 138. 39. 1/24 AS 4 138. 39/16 AS 2 138. 39. 1/24 AS 5 AS 2 thinks that all 138. 39/16 follow the 3 1 4 path 66 Advanced Networks

BGP Route Selection Process 1. 2. 3. 4. 5. 6. Maximum prefix length match Highest Local Priority Shortest AS Path Lowest MED (if routes through same AS) Min Cost Next hop router (consulting IGP) Prefer external to internal routes 1. Pick lowest BGP identifier among many E-BGP 2. Pick lowest BGP identifier among many I-BGP Advanced Networks 67

No Valley Prefer Customer Routing An abstraction of common sense policy • No valley = don’t transit traffic for peer or provider • Ie. If you don’t get paid, don’t do it ASes use the following rule, when selecting a path • Prefer a path through a customer • Prefer a path through a peer This policy can be implemented using BGP attributes for paths Advanced Networks 68

Advanced Networks 69

Internal BGP (I-BGP) Communication between routers of an AS I-BGP very similar to E-BGP except: • Different advertising rules Do not re-advertise a path internally • I-BGP 1 -> I-BGP 2 -X- I-BGP 3 But readvertise (in -out, out -in): • I-BGP 1 -> I-BGP 2 -> E-BGP 1 • E-BGP 1 -> I-BGP 2 -> I-BGP 1 Advanced Networks 70

I-BGP Re-Advertising I-BGP E-BGP A B D C B will not re-ad. to C what it hears from A But it will re-ad to D Why? Paths are identified by AS, and internally you have the same AS • To avoid routing loops Advanced Networks 71

I-BGP Mesh I-BGP routers form a fully connected mesh (clique) • Scalability becomes an issue The full mesh is independent of physical connectivity Advanced Networks 72

A Subtle Difference I-BGP vs E-BGP: routers communicate using the IP of the physical interface (link) • Link based reliability I-BGP: routers have “virtual or loopback” interface • Even when link fails, routers may be reachable • Node based reliability Advanced Networks 73

Static vs Dynamic Configuration In practice, many customers do not speak BGP Do not have an AS number They are configured statically Advanced Networks 74

One vs Many Providers Single homed vs multihomed Customers for reliability and performance connect to many providers Difference: • Single homed: easy to manage • Multihomed: tricky - Route aggregation - Load balancing Advanced Networks 75

Multihoming How can I use my multiple connections efficiently? Multihoming is quite widespread Users take it up to themselves to find reliability and good performance [Huston] Consequence: non-aggregatable state Advanced Networks 76

Two routers two links ISP 1 B C A Multiplex traffic at link or IP layer Use Next Hop to point to virtual router IP • Second look up: how can I reach virtual IP? • Pick one of the two links randomly or statically • Furthermore, when one link is down the other one is chosen Advanced Networks 77

Multihoming with One Routers 1 ISP 1 B C P 1 A P 2 Customer can advertise different prefixes on each link • Reliability? Use Multi-Exit-Discriminator • Cust. Sets MED, ISP uses it Use Local Preference • ISP sets LP and picks link • Customer’s IBGP to pick router Advanced Networks 78

Issues The mechanisms are there, but they depend on many factors • How much traffic each prefix generates? Balancing incoming and outgoing traffic Dynamically adapting to changing conditions Technical issues: • Some ISPs do not accept very long prefixes - Longer than they would be in classfull routing - Longer than 19 for new prefixes Advanced Networks 79

Multihoming with Two Routers 1 ISP Internet Similar case for Provider to Customer For Customer to Provider: ISP 1 C P 1 B ISP 1 Customers A P 2 • A could alternate paths - Reordering of packets • ISP 1 could advertise different addresses on each link - I. e. 1: ISP customers 2: Default (everybody else) Advanced Networks 80

Multihoming With Two Providers 138. 39/16 ISP 3 138. 39. 1/24 ISP 2 138. 39. 1/24 ISP 1 138. 39/16 138. 39. 1/24 Customer 138. 39. 1/24 Advanced Networks 81

Two Providers Multihoming: Getting Address Space Given two ISP Get space from one (we saw before) Get space from both • Advertise only related prefix to ISP - Aggregation but not reliability • Advertise both prefixes to ISPs - No aggregation but reliability Get space independently of both • Max flexibility, not reliability • If too narrow of prefix, may not propagate, no connectivity Advanced Networks 82

I-BGP Scalability Full mesh is not scalable: O(N^2) IBGP sessions Approaches to scalable I-BGP • Hierarchical structure: Route reflectors • Divide and conquer: Confederations Advanced Networks 83

Route Reflection D E B C A Explicitly allow some IBGP routers to readvertise Route reflectors: represent other routers Hierarchical structure avoids loops and problems Advanced Networks 84

Confederations D E B C A Decompose an AS to sub-AS Externally: one AS Internally: like E-BGP (E -I-BGP) Loop avoidance: • AS-CONFED-SET • AS-CONFED-SEQUENCE Advanced Networks 85

Confederation BGP Rules Differences of BGP between sub-ASes Local-Preference is transitive • Recall in BGP it is not NEXT-HOP attribute is forwarded Path within AS is monitored with sub-AS no. • AS-CONFED-SEQUENCE Advanced Networks 86

Handling the Dynamic Nature Advanced Networks 87

To Refresh or not to Refresh? BGP uses hard state: BGP routers consider a path is “usable” • until explicitly withdrawn • the session fails How do I detect if a connection failed? • Keep-alive messages Advanced Networks 88

Path Updates Frequency Send updates of a path no sooner 30 sec Why? • Stability • Overhead reduction Side-effects • Convergence can take longer What is the right interval? • Recent studies say that 30 s is too long Advanced Networks 89

Route Flapping and Dampening Flapping: constant alternating updates It can happen! Route update dampening: • Consider stability of path before using it How store “penalty” value for each path Issue: router needs to remember withdrawn paths Advanced Networks 90

Overview up to Now BGP is Distance Vector BGP uses TCP and hard-state Routing updates are “delayed” and batched Route dampening to alleviate instabilities Advanced Networks 91

End Advanced Networks 92

Current Research: the AS Topology Characterize the topology using power-laws (Faloutsos 99, Siganos et al 01) Modeling the evolution of the topology (Barabasi, Siganos 01) Advanced Networks 93

Ongoing Research: AS paths 107 Gb of AS paths over 3 years Exponential growth, but distances remain the same Inflation due to policy: • 20% of paths are larger than they could Significant Routing Asymmetry: • 40% paths by at least one hop Advanced Networks 94

Near Future Research Plans Mine the collected paths for patterns How stable were the paths? Can we identify “illegal” paths? Identify pathologies (ie. loops) Advanced Networks 95

Overview of Plans for Research How well does BGP work now? How secure and robust is BGP? • Cyber-terrorism: how much damage can be done? How would we design BGP now? • People are asking this question How well will BGP scale? How can we manage BGP (avoid human errors)? Approach: • Analytical and simulations with SSFNET Advanced Networks 96

Conclusions BGP is an open and exciting topic The community knows very little Big ticket items: • • Measurements and modeling Robustness, security Network Management: traffic engineering Scalability Advanced Networks 97

Multihoming With Two Providers 138. 39/16 ISP 3 138. 39. 1/24 ISP 2 138. 39. 1/24 ISP 1 138. 39/16 138. 39. 1/24 Customer 138. 39. 1/24 Advanced Networks 98

Policies on Transitivity Isp 1 Isp 2 Nontransit A customer should not Be transit for its Providers Not allowed AS X Transit AS 1 AS 3 AS 2 Advanced Networks AS 4 99

More BGP attributes • Communities • Confederations • Route Reflectors Advanced Networks 100

Convergence Advanced Networks 101

Multi-Exit Discriminator(MED) • Indication to external peers of the preferred path into the AS • Lowest Med Preferred Advanced Networks 102

Note in MED One AS sets the values Another AS interpets and uses them Thus: • Cooperative ASes • Only between two ASes (1 hop scope) MED is meaningless in the next hop MED can be used only if both routes are advertised from the same AS Advanced Networks 103

Local Preference Choosing paths internally Within an AS: • Set Local Preference to pick the path you want to send data to The higher Local Preference is preferred Advanced Networks 104

Atomic Aggregate Sometimes we aggregate paths that are different (ie different AS sequence) Atomic aggregate shows that some of the destinations in this prefix are not necessarily following the same path Advanced Networks 105

Aggregator Notify that an aggregation took place • Which AS • Which router For management and traceback purposes Advanced Networks 106

Example: The Intended Use Provider Peer Advanced Networks Customer Peer 107

BGP Graph and Routing Policies 200 10 11 1 2 12 3 13 4 Up then down: 1, 100, 200, 13, 4 No valleys, no up-down-up, no more than 1 peer-peer Advanced Networks 108