Donald E Eastlake 3 rd 1 CoChair TRILL

Donald E. Eastlake, 3 rd 1 Co-Chair, TRILL Working Group Principal Engineer, Huawei d 3 e 3 e 3@gmail. com TRILL - SVC COMSOC Transparent Interconnection of Lots of Links April 2013 Overview of TRILL

Donald E. Eastlake, 3 rd Principal Engineer at Huawei Technologies Formerly with Cisco and before that Motorola. Co-Chair of the IETF TRILL Working Group TRILL - SVC COMSOC April 2013 Chair of the IETF PPPEXT Working Group Chair of the IEEE 802. 11 ak Task Group Author of 59 IETF RFCs. Note: This tutorial represents my personal views, not those of the TRILL WG or Huawei. It is a high level technical overview. It is not practical to include all the details in the specification documents in a presentation of this length. 2

CONTENTS 3. 4. 8. 9. 10. 11. TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures 3

WHAT IS TRILL? April 2013 TRansparent Interconnection of Lots of Links TRILL - SVC COMSOC WG Charter http: //www. ietf. org/dyn/wg/charter/trill-charter. html Standardized by IETF TRILL Working Group: Donald E. Eastlake 3 rd (Huawei), Co-Chair Erik Nordmark (Cisco), Co-Chair Jon Hudson (Brocade), Secretary TRILL Switch / Router / RBridge (Routing Bridge) Device that implements TRILL/RBridge Campus A network of RBridges, links, and any intervening bridges, bounded by end stations / layer 3 routers. 4

WHAT IS TRILL? April 2013 A Compatible Protocol end nodes just think it is Ethernet. The more bridges you convert to TRILL switches, the better your network’s stability and bandwidth utilization. Terminates Spanning Tree Protocols TRILL - SVC COMSOC Attached 5

INSPIRED BY A REAL LIFE INCIDENT April 2013 TRILL - SVC COMSOC In November 2002, Beth Israel Deaconess Hospital in Boston, Massachusetts, had a total network meltdown: Their network took four days of heroic efforts to be restored to an operational state! In the mean time the staff was reduced to using paper and pencil. Beth Israel Deaconess had grown by acquiring various clinics and just plugged all those bridged networks together. The article in Boston’s primary newspaper specifically mentioned “Spanning Tree Protocol” as the problem! Radia Perlman, who invented spanning tree over 25 years ago, decided it was time to come up with a better way. 6

WHAT IS TRILL? April 2013 Basically a simple idea: native Ethernet frames in a transport header providing a hop count Route the encapsulated frames using IS-IS Decapsulate native frames before delivery TRILL - SVC COMSOC Encapsulate Provides Least cost paths with zero/minimal configuration Equal Cost Multi-Pathing of unicast traffic Multi-pathing of multi-destination traffic 7

CONTENTS 3. 4. 8. 9. 10. 11. TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures 8

UNICAST LEAST COST PATHS April 2013 B 3 A three bridge network TRILL - SVC COMSOC = end station B 1 B 2 9

UNICAST LEAST COST PATHS April 2013 B 3 TRILL - SVC COMSOC = end station B 1 B 2 Spanning tree eliminates loops by disabling ports 10

UNICAST LEAST COST PATHS April 2013 RB 3 TRILL - SVC COMSOC = end station RB 1 RB 2 A three RBridge network: better performance using all facilities 11

UNICAST MULTI-PATHING April 2013 B 1 B 5 B 2 TRILL - SVC COMSOC B 3 B 4 = end station Bridges limit traffic to one path 12

UNICAST MULTI-PATHING April 2013 RB 1 RB 5 RB 2 TRILL - SVC COMSOC RB 3 RB 4 RBridges support multi-path for higher throughput = end station 13

TRILL FEATURES Multi-tenant support Frame Priorities Data Center Bridging Virtualization Support Multi-pathing Optimal Paths Rapid Fail Over The safety of a TTL TRILL - SVC COMSOC Transparency Plug & Play Virtual LANs Routers April 2013 Bridges TRILL Switch Implemented in data plane Extensions 14

MORE TRILL FEATURES Transit RBridges do not learn end station addresses. Compatible with existing IP Routers. TRILL switches are as transparent to IP routers as bridges are. MTU feature and jumbo frame support including jumbo routing frames. Has a poem. TRILL - SVC COMSOC Breaks up and minimizes spanning tree for greater stability. Unicast forwarding tables at transit RBridges scale with the number of RBridges, not the number of end stations. April 2013 The only other bridging or routing protocol with a poem is Spanning Tree (see Algorhyme). 15

Algorhyme V 2 (TRILL) - By Ray Perlner (Radia Perlman’s son) TRILL - SVC COMSOC I hope that we shall one day see A graph more lovely than a tree. A graph to boost efficiency While still configuration-free. A network where RBridges can Route packets to their target LAN. The paths they find, to our elation, Are least cost paths to destination! With packet hop counts we now see, The network need not be loop-free! RBridges work transparently, Without a common spanning tree. April 2013 16

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 17

A A A GB CB GB A A A CB Spanning Tree Eliminates Loops by Disabling Ports GB A A Acme Power Plant A A = Access Bridge GB = a. Greggation Bridge CB = Core Bridge TRILL - SVC COMSOC 1. Rapid Spanning Tree Protocol Domain A April 2013 GB A 18

A A GB GB RSTP Domain A A A CRB A A GB A A RSTP Domain 1. Acme Power Plant Process Control Network with RBridge Core breaking up spanning tree GB A A TRILL - SVC COMSOC CRB GB A April 2013 RSTP Domain A A A = Access Bridge GB = a. Greggation Bridge CRB = Core RBridge 19

A A A GRB A A April 2013 GRB A A GRB Power Plant GRB A A A Process Control Network with RBridge Mesh eliminating spanning tree GRB A A TRILL - SVC COMSOC 1. Acme A A = Access Bridge GRB = a. Greggation RBridge 20

1: 1 Backup B B B Dist. Bridge B B B = Head of Rack Bridge B B TRILL - SVC COMSOC B Wan Router April 2013 Distribution Bridges must be able to handle 100% of the load. Only 1 path available between any pair of “B”s. Wan Router 2. Acme Data Center Network with Bridges B 21

N: 1 Backup Dist. RBridge Wan Router Dist. RBridge H = Head of Rack RBridge H H H H TRILL - SVC COMSOC Dist. RBridge Wan Router April 2013 Distribution RBridges need to handle only 25% of the load. Multiple available paths between “H”s. 2. Acme Data Center Network with RBridges H 22

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 23

TRILL PRODUCTS products Cisco Some TRILL Standard Products Announced on the Web Broadcom – Strata. XGS Trident (BMC 5680) Cisco – Nexus, Catalyst 6500 IBM / Blade Networks – Rack. Switch G 8264 HP – 5900 To. R Switches Huawei – Cloud Engine 5800, 6800, 12800 Mellanox – Switch. X ZTE – ZXR 10 5800 Series TRILL - SVC COMSOC Fabric. Path Brocade VCS April 2013 “Pre-standard” 24

TRILL SILICON Here are six publicly known independent silicon are multiple different chips. • Broadcom – merchant silicon • Brocade – products • Cisco – products • Fulcrum – merchant silicon • Marvell – merchant silicon • Mellanox – merchant silicon TRILL - SVC COMSOC implementations of the TRILL Fast Path. In some cases there April 2013 • 25

TRILL OPEN SOURCE Oracle: TRILL for Solaris • http: //hub. opensolaris. org/bin/view/Project+rbridges/Web. Home TRILL Port to Linux: National University of Sciences and Technology (NUST), Islamabad, Pakistn TRILL - SVC COMSOC April 2013 • Muhammad Mohsin Sardar mohsin. sardar@seecs. edu. pk http: //wisnet. seecs. nust. edu. pk/projects/trill/index. html 26

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 27

TRILL IS BASED ON IS-IS runs directly at Layer 2. Thus no IP addresses are needed, as they are for OSPF, and IS-IS can run with zero configuration. IS-IS uses a TLV (type, length, value) encoding which makes it easy to define and carry new types of data. TRILL - SVC COMSOC The IS-IS (Intermediate System to Intermediate System) link state routing protocol was chosen for TRILL over IETF OSPF (Open Shortest Path First), the only plausible alternative, for the following reasons: April 2013 28

TRILL NICKNAMES April 2013 TRILL - SVC COMSOC TRILL switches are identified by IS-IS System ID and by 2 -bytes nicknames. Nicknames can be configured but by default are auto-allocated. In case of collisions, the lower priority RBridge must select a new nickname. Nicknames: Saves space in headers. An RBridge can hold more than one nickname so that It can be the root of more than one different distribution tree. May be used to distinguish frames following traffic engineered routes versus least cost routes. 29

TRILL ENCAPSULATION AND HEADER On Ethernet links, the link header is addressed from the local source RBridge to the next hop RBridge for known unicast frames or to the All-RBridges multicast address for multi-destination frames. The TRILL header specifies the first/ingress RBridge and either the last/egress RBridge for known unicast frames or the distribution tree for multi-destination frames. TRILL - SVC COMSOC TRILL Data frames between RBridges are encapsulated in a TRILL Header and then in a local link header. April 2013 30

TRILL ENCAPSULATION AND HEADER Reasons for encapsulation: Provides TRILL - SVC COMSOC a hop count to mitigate loop issues To hide the original source address to avoid confusing any bridges present as might happen if multi-pathing were in use To direct unicast frames toward the egress RBridge so that forwarding tables in transit RBridges need only be sized with the number of RBridges in the campus, not the number of end stations To provide a separate outer VLAN tag, when necessary, forwarding traffic between RBridges, independent of the original VLAN of the frame April 2013 31

TRILL HEADER DETAILS April 2013 TRILL Header Egress RBridge Nicknames V R M Ext. Lng Hop Ingress RBridge Nickname – auto-configured 16 -bit campus local names for RBridges V = Version (2 bits) R = Reserved (2 bits) M = Multi-Destination (1 bit) Ext. Lng = Length of TRILL Header Extensions Hop = Hop Limit (6 bits) TRILL - SVC COMSOC TRILL Ethertype 32

TRILL OVER ETHERNET DA DA VLAN* TRILL Ethertype 0 x 22 F 3 TRILL Header Payload Frame (DA, SA, VLAN/Tenant, Data) Original Frame with VLAN/Tenant Label Ethernet Link Transport Header SA VLAN* RBridge One FCS TRILL - SVC COMSOC IS-IS: SA L 2 -IS-IS Ethertype 0 x 22 F 4 IS-IS PDU FCS RBridge Two Ethernet Cloud * Link Transport VLAN only needed for VLAN sensitive link. April 2013 Data: RBridge Three 33

TRILL OVER PPP TRILL Data 0 x 005 D TRILL Header PPP FCS Original Frame with VLAN/Tenant Label IS-IS: PPP TRILL IS-IS 0 x 405 D RBridge One PPP FCS IS-IS PDU PPP TRILL - SVC COMSOC PPP Link Transport Header Payload Frame (DA, SA, VLAN/Tenant, Data) April 2013 Data: RBridge Two 34

MAC ADDRESS LEARNING By default, TRILL switches at the edge (directly connected to end stations) learn attached VLAN/MAC addresses from data. Optionally, MAC addresses can be passed through the control plane. MAC addresses can be statically configured. Transit TRILL switches do not learn end station addresses. TRILL - SVC COMSOC By IS-IS all TRILL switches in the campus learn about and can reach other but what about reaching end station MAC addresses? April 2013 35

RBRIDGES & ACCESS LINKS April 2013 TRILL - SVC COMSOC You can have multiple TRILL switches on a link with one or more end stations. One is elected to be in charge of the link and to handle end station traffic. But to load split, it can assign VLANs to other RBridges on the link. TRILL Cloud RB 1 RB 2 RB 3 B 1 B 2 B 3 36

MULTI-DESTINATION TRAFFIC Multi-destination data is sent on a bi-directional distribution tree. root of a tree is a TRILL switch or a link (pseudonode) determined by a separate election and represented by nickname. The ingress RBridge picks the tree, puts the tree root nickname in the “egress nickname” slot, and sets the M bit in the TRILL Header. TRILL - SVC COMSOC The April 2013 All the TRILL switches in a campus calculate the same trees. All trees reach every TRILL switch in the campus. 37

MULTI-DESTINATION TRAFFIC There are no end stations downstream in the VLAN of the frame, or The frame is multicast and there is no multicast listener or multicast router downstream. TRILL - SVC COMSOC As a frame is propagated on a distribution tree, its distribution can be pruned by VLAN and by multicast group since it is not useful to send a frame down a tree branch if April 2013 38

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 39

PEERING: ARE RBRIDGES OR ROUTERS? Really, they are a new species, between IEEE 802. 1 bridges and Layer 3 routers: (plus servers and other end stations) TRILL Switches TRILL - SVC COMSOC Layer 3 Routers April 2013 Bridges Hubs/Repeaters 40

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 41

FINE GRAINED LABELING native frame VLAN and input port into a fine grained label on ingress and do the reverse mapping on egress. Remember the priority and DEI of native frames on ingress and restores them on egress. Fine Grained Label TRILL switches are a superset of a base protocol TRILL switches. They support VLANs as in the base standard on a port if not configured to do Fine Grained Labeling. TRILL - SVC COMSOC Map April 2013 With Fine Grained Labeling (FGL), the inner data label is an extended 24 -bit label. Ingress/egress TRILL switches that support FGL: 42

FINE GRAINED LABELING April 2013 From the current draft: https: //datatracker. ietf. org/doc/draft-ietf-trill-fine-labeling/ Link Header DA SA TRILL Header DA SA 8100 Pri, DEI VID Payload Frame Data Link Trailer Data Label Fine Grained Labeling: Link Header TRILL - SVC COMSOC Payload Base protocol VLAN Labeling: 893 B Transit Priority Pri, DEI Upper 893 B Pri, DEI Lower 24 -Bit Tenant ID Payload Frame Data Ingress/Egress Priority Link Trailer 43

FINE GRAINED LABELING MIGRATION An initial deployment of VLAN labeling TRILL switches can be smoothly extended to Fine Grained Labeling VL TR 3 Fine Grained Labeling TRILL Switches FGL TR 4 TRILL - SVC COMSOC VLAN Labeling TRILL Switches April 2013 FGL TR 6 VL TR 1 VL TR 2 FGL TR 5 44 VLAN Labeled Path FGL Labeled Path

FINE GRAINED LABELING MIGRATION Some VL TRILL switches are convertible to FGL-safe RBridges (FGL transit only) with a software upgrade. TRILL - SVC COMSOC Fine Grained Labeling TRILL Switches FGL TR 3 FGL TR 1 FGL TR 2 April 2013 FGL TR 4 FGL TR 6 FGL TR 5 45 VLAN Labeled Path FGL Labeled Path

FINE GRAINED LABELING MIGRATION Even if not upgradable, they can generally be connected. VLAN Labeling TRILL Switches VL TR 3 Fine Grained Labeling TRILL Switches FGL TR 4 TRILL - SVC COMSOC Some VL TRILL switches are convertible to FGL-safe RBridges (FGL transit only) with a software upgrade. April 2013 FGL TR 6 VL TR 1 VL TR 2 FGL TR 5 46 VLAN Labeled Path FGL Labeled Path

CONTENTS 3. 4. 8. 9. 10. 11. How TRILL Works Peering and Layers Fine Grained Labeling Status and Futures TRILL - SVC COMSOC 2. What is TRILL? TRILL Features Two TRILL Examples TRILL Products April 2013 1. 47

STANDARDIZATION STATUS TRILL protocol RFCs (bold = stds track) 5556, “TRILL Problem and Applicability” RFC 6325, “RBridges: TRILL Base Protocol Specification” RFC 6326, “TRILL Use of IS-IS” RFC 6327, “RBridges: Adjacency” RFC 6361, “TRILL over PPP” RFC 6439, “RBridges: Appointed Forwarders” RFC 6847, “FCo. E over TRILL” RFC 6850, “Definitions of Managed Objects for RBridges” (MIB) RFC 6905, “TRILL OAM Requirements” TRILL - SVC COMSOC RFC April 2013 The 48

STANDARDIZATION STATUS l “TRILL: Clarifications, Corrections, and Updates” l l “TRILL: BFD Support” l l https: //datatracker. ietf. org/doc/draft-ietf-trill-rbridge-bfd/ “TRILL: RBridge Channel Support” l l https: //datatracker. ietf. org/doc/draft-ietf-trill-clear-correct/ TRILL - SVC COMSOC Document that are fully approved and in the RFC Editor’s Queue. These are expected to issue as standards track RFCs soon: April 2013 l https: //datatracker. ietf. org/doc/draft-ietf-trill-rbridge-channel/ “TRILL: Header Extension” l https: //datatracker. ietf. org/doc/draft-ietf-trill-rbridge-extension/ 49

OAM DOCUMENTS STATUS TRILL - SVC COMSOC draft-ietf-trill-rbridge-bfd-07. txt draft-ietf-trill-rbridge-channel-08. txt April 2013 SNMP RFC 6850, “Definitions of Managed Objects for RBridges” (MIB) BFD over TRILL In RFC Editor’s queue: TRILL OAM (carrier grade) RFC 6905: TRILL OAM Requirements draft-ietf-trill-oam-framework-01. txt (Framework) draft-tissa-trill-oam-fm-01. txt (Fault Management) draft-mizrahi-trill-loss-delay-00. txt (Performance) 50

MORE TRILL REFERENCES TRILL Introductory Internet Protocol Journal Article: http: //www. cisco. com/web/about/ac 123/ac 147/archived_issues/ ipj_14 -3/143_trill. html The initial paper: Perlman, Radia. “Rbridges: Transparent Routing”, Proceeding Infocom 2004, March 2004. TRILL - SVC COMSOC April 2013 http: //www. ieee-infocom. org/2004/Papers/26_1. PDF 51

TRILL FUTURES l l l In data centers, the location of all MAC and IP address and virtual machines is typically known. Both Push and Pull directory services to eliminate most ARP (IPv 4), ND (IPv 6), and unknown unicast MAC flooding. Carrier grade OAM Active-active at the edge Multi-level and Multi-topology support 3 rd TRILL Plugfest planned at UNH IOL the week of May 20 th TRILL - SVC COMSOC l Directory Assisted Edge: April 2013 l 52

53 Co-Chair, TRILL Working Group Principal Engineer, Huawei d 3 e 3 e 3@gmail. com TRILL - SVC COMSOC Donald E. Eastlake 3 rd April 2013 END

April 2013 Donald E. Eastlake 3 rd 54 Co-Chair, TRILL Working Group Principal Engineer, Huawei d 3 e 3 e 3@gmail. com TRILL - SVC COMSOC Backup Slides

TRILL HISTORY UP TO 2009 l l l l l TRILL - SVC COMSOC l 1964: Packet switching/routing invented by Paul Baran. 1973: Ethernet invented by Robert Metcalfe 1979: Link State Routing invented by John Mc. Quillan. 1985: Radia Perlman invents the Spanning Tree Protocol. 1987: DECnet Phase V / IS-IS designed by Radia Perlman. 2002: Beth Israel Deaconess Hospital network in Boston melts down due to deficiencies in the Spanning Tree Protocol. 2004: TRILL invented by Radia Perlman, presented at Infocom. 2005: TRILL presented to IEEE 802 by Radia Perlman, rejected. 2005: IETF Charters the TRILL Working Group. 2008: MTU problem delays protocol while fix is incorporated. 2009: RFC 5556 “TRILL: Problem and Applicability Statement” 2009: TRILL Protocol passed up to IESG for Standards Approval. April 2013 l 55

TRILL IN 2010 TO 2013 SO FAR 2011: TRILL Protocol base document set published: l RFC 6325: “RBridges: TRILL Base Protocol Specification” (Includes TRILL over Ethernet) l RFC 6326: “TRILL Use of IS-IS” l RFC 6327: “RBridges: Adjacency” l RFC 6361: “TRILL over PPP” l RFC 6439: “RBridges: Appointed Forwarders” l 2012: 2 nd TRILL plugfest at UNH IOL l 2013: More TRILL Progress so far l RFC 6847: “FCo. E over TRILL” l RFC 6850: “Definition of Managed Objects for RBridges” l RFC 6905: “TRILL OAM Requirements” TRILL - SVC COMSOC l April 2013 l 2010: TRILL approved as IETF Standard (15 March 2010) 2010: Successful TRILL control plane plugfest at UNH IOL l 56

INPUT PORT PROCESSING Detailed example of unicast frame TRILL routing on an Ethernet link Dest MAC Data Src MAC FCS VLAN TRILL - SVC COMSOC Input Native Frame on link: April 2013 Input port adds VLAN-ID and priority if frame untagged Input Native Frame after input port: Dest MAC Src MAC VLAN Data FCS 57

TRILL UNICAST INGRESS Dest MAC Src MAC VLAN DA SA VLAN 1 Link Transport Header TTL=n FCS TRILL - SVC COMSOC Look Up Egress, Next Hop DA & Output Port Data April 2013 Input Native Frame: Output TRILL Data Frame: Egress Ingress TRILL Header Ingressing RBridge Payload Frame New FCS Original Frame with VLAN or Tenant ID 1 Outer VLAN tag is a transport artifact and only needed if RBridges are connected by a bridged LAN or carrier Ethernet requiring a VLAN tag or the like. 58

TRILL UNICAST TRANSIT Input TRILL Data Frame: SA Transit RBridge DA SA VLAN 1 TRILL Hdr Egress TTL=n Ingress Payload Frame FCS Payload Frame New FCS Look Up Next DA & Output Port VLAN 1 TRILL Hdr Egress TTL=n-1 Ingress TRILL - SVC COMSOC DA April 2013 Incoming Link Transport Header Output TRILL Data Frame: Outgoing Link Transport Header 1 Input and output Outer VLANs can differ. The true VLAN or 59 Tenant ID of the data is inside the payload frame. Outer VLAN only needed if link is VLAN sensitive.

TRILL UNICAST EGRESS SA TRILL Hdr VLAN 1 Egress Input TRILL Data Frame: Payload Frame FCS Output Native Frame: Dest MAC Src MAC VLAN 2 Data TRILL - SVC COMSOC DA Egressing RBridge April 2013 Link Transport Header New FCS Look Up Output Port 1 Outer VLAN only needed if RBridges are connected by a bridged LAN or carrier Ethernet requiring a VLAN tag or the like 2 Final native frame VLAN tag may be omitted depending on RBridge 60 output port configuration.

OUTPUT PORT PROCESSING April 2013 Output Native Frame before output port: Src MAC VLAN Data New FCS Output port may be configured to output untagged and will do so by default for the port VLAN ID Dest MAC Src MAC Data TRILL - SVC COMSOC Dest MAC New FCS 61