Elmo Muhammad Shahbaz Lalith Suresh Jennifer Rexford Nick

Elmo Muhammad Shahbaz Lalith Suresh, Jennifer Rexford, Nick Feamster, Ori Rottenstreich, and Mukesh Hira 1

Elmo: Source Routed Multicast for Public Clouds Muhammad Shahbaz Lalith Suresh, Jennifer Rexford, Nick Feamster, Ori Rottenstreich, and Mukesh Hira 2

1 -to-Many Communication in Cloud 3

1 -to-Many Communication in Cloud Distributed Programming Frameworks State Replication Streaming Telemetry Publish-Subscribe Systems Infrastructure Applications and more … 4

1 -to-Many Communication in Cloud 10, 000 s of tenants 100 s of workloads Millions of groups 5

1 -to-Many Communication in Cloud 10, 000 s of tenants 100 s of workloads Millions of groups Multicast 6

1 -to-Many Communication in Cloud 10, 000 s of tenants 100 s of workloads Millions of groups Multicast 7

Limitations of Native Multicast Controller 8

Limitations of Native Multicast Processing overhead Controller Excessive control churn due to membership and topology changes Limited group entries 9

Restricted to Unicast-based Alternatives Controller Processing overhead S R R 10

Restricted to Unicast-based Alternatives Controller Traffic overhead Processing overhead S R R 11

1 -to-Many Communication in the Cloud Controller S R R 12

1 -to-Many Communication in the Cloud Controller Traffic overhead Processing overhead S R R 13

1 -to-Many Communication in the Cloud Processing overhead Controller Excessive control churn due to membership and topology changes Traffic overhead Processing overhead S R Limited group entries R 14

1 -to-Many Communication in the Cloud Processing overhead Controller Excessive control churn due to membership topology changes Need a scheme that scales to millions of and groups without excessive control, end-host CPU, and traffic overheads! Traffic Limited group entries overhead Processing overhead S R R 15

Proposal: Source Routed Multicast Controller S R R 16

Proposal: Source Routed Multicast Controller S R R 17

Proposal: Source Routed Multicast Little processing overhead Controller Minimal control churn No group entries needed No traffic overhead Negligible processing overhead S R R 18

A Naïve Source Routed Multicast A multicast group encoded as a list of (Switch, Ports) pairs For a data center with: - 1000 switches - 48 ports per switch Switch 1: [Ports ] Switch 2: [. . . ] O(30) bytes per switch Switch 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 20 x the packet size! O(30, 000) bytes header for a group spanning 1000 switches 19

Enabling Source Routed Multicast in Public Clouds Key attributes: - Efficiently encode multicast forwarding policy inside packets - Process this encoding at hardware speed in the switches - Execute tenants’ applications without modification 20

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Ports ] Switch 2: [. . . ] Switch 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 21

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs 1 Encode switch ports as a bitmap Switch 1: [Bitmap] Switch 2: [. . . ] Switch 3: [. . . ] Switch 4: [. . . x. . ] Bitmap is the internal data structure that switches use for replicating packets Switch 5: [. x. . . ] 22

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs 2 Group switches into layers Switch 1: [Bitmap] Switch 2: [. . . ] Switch 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 23

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2: [. . . ] Switch 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 2 Group switches into layers Core Spine Leaf 24

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs 3 Switches within a layer with same ports share a bitmap Switch 1: [Bitmap] Switch 2: [. . . ] Switch 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 25

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 3 Switches within a layer with same ports share a bitmap Core Spine Leaf 26

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs ports share a bitmap Core Switch 2, 3: [. . . ] Spine Modern commodity switches can parse packet headers of 512 bytes Switch 4: [. . . x. . ] Leaf Switch 5: [. x. . . ] No. of groups Switch 1: [Bitmap] 3 Switches within a layer with same For a data center with: - 628 switches - 325 bytes header space Supports 890, 000 groups! 27

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 3 Switches within a layer with same ports share a bitmap Core Spine Leaf 28

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4: [. . . x. . ] Switch 5: [. x. . . ] 4 Switches within a layer with N different ports share a bitmap Core Spine Leaf 29

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] 4 Switches within a layer with N different ports share a bitmap Core Spine Leaf 30

Encoding a Multicast Policy in Elmo A multicast group encoded as a list of (Switch, Ports) pairs Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] different ports share a bitmap Core Spine Leaf No. of groups Switch 1: [Bitmap] 4 Switches within a layer with N For a data center with: - 628 switches - 325 bytes header space Supports 980, 000 groups! Difference in ports 31

Encoding a Multicast Policy in Elmo Fixed Header Size A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] 4 Switches within a layer with N different ports share a bitmap Core Spine Leaf 32

Encoding a Multicast Policy in Elmo Fixed Header Size A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] 5 Use switch entries and a default bitmap for larger groups Core Spine Leaf Default Bitmap Switch Table Entries 33

Encoding a Multicast Policy in Elmo Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] Default Bitmap Switch Table Entries bitmap for larger groups Core Spine Leaf Switch entries Switch 1: [Bitmap] 5 Use switch entries and a default For a data center with: - 628 switches - 325 bytes header space Traffic overhead Fixed Header Size A multicast group encoded as a list of (Switch, Ports) pairs Difference in ports 34

Encoding a Multicast Policy in Elmo Fixed Header Size A multicast group encoded as a list of (Switch, Ports) pairs Switch 1: [Bitmap] Switch 2, 3: [. . . ] Switch 4, 5: [. x. . ] Default Bitmap Switch Table Entries 1 Encode switch ports as a bitmap 2 Group switches into layers Core Spine Leaf Switches within a layer with: 3 - same ports share a bitmap 4 - N different ports share a bitmap Use switch entries and a default 5 bitmap for larger groups For a data center with: - 628 switches - 325 bytes header space Supports a Million groups! 35

Processing a Multicast Policy in Elmo 2. Computes the multicast policy 1. API Controller 3. Installs entries in programmable - virtual switches to push Elmo headers on packets - hardware switches - More flow entries and higher update rates than hardware switches - No changes to the tenant application Virtual Switch S R R 36

Processing a Multicast Policy in Elmo Controller Switch looks for: Matching bitmap or Table entry or Default bitmap S R R 37

Publisher Subscriber Applications Run Without Performance Overhead 38

Conclusion Elmo • Designed for multi-tenant data centers • Compactly encodes multicast policy inside packets Source Routed Multicast for Public Clouds • Operates at hardware speed using programmable data planes Learn more here: https: //elmo-mcast. github. io 39

40

Backup Slides 41

Control Plane Scalability For a multi-rooted Clos topology with 27 K hosts and p-rule header of 325 bytes: * min (max) * https: //conferences. sigcomm. org/co-next/2013/program/p 61. pdf 42

Elmo Adds Negligible Overheads to Soft. Switches 43

Elmo Operates within the Header Size Limit of Switch ASICs For a 256 -port, 200 mm 2 baseline switching ASIC that can parse a 512 -byte packet header: 190 bytes for other protocols (e. g. , datacenter protocols take about 90 bytes) 44

Elmo’s Primitives are Inexpensive to Implement in Switch ASICs For a 256 -port, 200 mm 2 baseline switching ASIC that can parse a 512 -byte packet header: As a comparison, Conga consumes 2% of area and Banzai consumes 12% of area. 45