Expanders as Datacenter Topologies Michael Dinitz Michael Schapira

Expanders as Datacenter Topologies Michael Dinitz Michael Schapira Gal Shahaf Asaf Valadarsky Xpander: Towards Optimal-Performance Datacenters. Co. Next ’ 16 Asaf Valadarsky, Gal Shahaf, Michael Dinitz, Michael Schapira Large Fixed-Diameter Graphs are Good Expanders. ESA ‘ 18 Michael Dinitz, Michael Schapira, Gal Shahaf

Designing a Datacenter Architecture Network topology? Routing? Congestion control?

Designing a Datacenter Architecture Performance ➡Throughput ➡Resiliency to failures ➡Path diversity ➡Flow completion time ➡… Deployability ➡Cabling complexity ➡Operations cost ➡Equipment costs ➡“Easy to reason about” ➡…

Performance What is the “right” architecture? ? ? Jellyfish Slim-Fly SWDC, DCell, BCube, c-Through, Helios, … Fat Tree Deployability

Today • Reaching the upper-right corner through “expander datacenters” • Co. Next ‘ 16: A tangible, near-optimal datacenter design (Xpander) • ESA ‘ 18: Even other approaches (Slim-Fly, “degree-diameter graphs”) are expanders in disguise! • Message: if you’re designing datacenters, think about using expanders!

Expanders and Xpander: Towards Optimal-Performance Datacenters. Co. Next ’ 16 Asaf Valadarsky, Gal Shahaf, Michael Dinitz, Michael Schapira

Expander Graphs • Intuition: large cuts, so traffic never gets “trapped”. • Edge Expansion: X • Expander: d-regular graph with expansion �� (d) • The closer to d/2 the better G

Performance: Throughput • One measure of performance: throughput • Model network as capacitated graph G = (V, E, c), demand matrix D • Throughput: max concurrent flow • Largest �� so that every commodity can send �� fraction of its demand, obeying capacities • Argued by Jyothi et al. [SIGMETRICS ‘ 14] to be a good metric. • Important selections of D: • Arbitrary, all-to-all, random permutation, one-to-many (multicast), …

Expanders: Throughput Theorem: for all-to-all demands, any expander has throughput within O(log d) of the best possible d-regular graph Theorem: for any demands D, any expander has throughput within O(log n) of the best possible d-regular graph (for that D) Theorem: for any d-regular graph G, there is some D and d-regular graph G* such that G* has throughput �� (log n) more than G.

In practice**** • Constants matter! • What about deployability? • Jellyfish [NSDI ‘ 12]: random graphs (expanders) • Great performance! • Tricky deployment • Xpander: construction based on Bilu-Linial expander • Same performance as Jellyfish, easier deployment

Xpander Datacenter Architecture No links within the same metanode To. R To. R Same number of links between every two metanodes Same number of To. Rs withina deterministic graph-theoretic Leverages any metaconstruction of expanders [BL ’ 06] To. R

COST EFFICIENCY: XPANDER VS. FAT-TREE Switch Degree #Switches All-to-All Throughput 8* 80% 121% 10 100% 157% 24 80% 111% *Validated using Mininet experiments

No links within the same metanode Cabling Xpander Same number of links between every two metanodes ➡Switch Place To. Rs of each meta-node in close #Switches #Servers #Cables Cable Length Throughput Ports proximity 42 vs. 48 504 vs. 512 420 vs. 512 4. 2 km vs 5. 12 km 32 109% ➡ Bundle cables between two meta-nodes (87. 5%) (98. 44%) (82%) ➡ Use color-coding to distinguish between 10. 5 km vs 66 vs. 72 1056 vs. 1152 48 different meta-nodes and bundles of cables 11. 5 km 142% (92%)

Cabling Xpander

Conclusions (Part I) • Expanders are good datacenter topologies in theory and in practice • Intuitive obstacles (cabling complexity and length) can be overcome by using the right expander • Tons of open questions! • Combining with programmable/reconfigurable topologies? • Kassing et al. , “Beyond fat-trees without antennae, mirrors, and disco-balls”, SIGCOMM ’ 17 • Non-regular / heterogeneous datacenters?

Expanders vs. Degree-Diameter Large Fixed-Diameter Graphs are Good Expanders. ESA ‘ 18 Michael Dinitz, Michael Schapira, Gal Shahaf

Different Intuitions • Expanders • Good because data can’t get “bottlenecked” anywhere • Ensure this by making cuts “large” • Alternative • Long paths are wasteful: flow of size �� uses (length × �� ) total capacity • So minimize distances: try to make diameter small

Degree-Diameter Problem • Three parameters: size n, degree d, diameter k • What are the extremal graphs? • “Degree/Diameter problem” • “Intuitively, the best known degree-diameter topologies should support a large number of servers with high network bandwidth and low cost (small degree). . . Thus, we propose the best-known degree-diameter graphs as a benchmark for comparison. ” — Singla et al, NSDI ’ 12 • Slim Fly [Besta-Hoefler, SC ’ 14]: Uses near-optimal degree-diameter graphs for k=2 (MMS graphs) and k=3 (BDF and Delorme graphs)

Network design Expander graphs Low diameter networks [Es 85, Be 89, MMS 98, Ki 07, Ba 10, Le 14] ? Q: What are the relationships between them?

Main Question Example: Question: Do low-diameter graphs admit good expansion properties? Low Diameter ? Expansion • In general: NO. • However, most low-diameter network architectures are good expanders. • Why?

Main Question (refined) Question: Do large low-diameter graphs admit good expansion properties? Answer: Yes! Low Diameter Large Expansion So degree-diameter graphs are just a special case of expanders! Experimental results: not quite as good as Xpander, Jellyfish

How large can a graph possibly be? •

Results (high level) • Algebraic / spectral expansion: �� (G) = second largest eigenvalue of adj. matrix • Cheeger inequality: h(G) ≥ (d-�� (G))/2 Theorem: Any (d, k) graph G with n ≥ μd, k – O(dk/2) has λ(G) ≤ O(d 1/2) Theorem: Any (d, k) graph G with n ≥ (1 -ε)μd, k has λ(G) ≤ O(ε 1/k) × d

Results

Techniques • Connection to Geronimus Polynomials • Formal polynomials Pt(x), where Pt(A) is matrix giving # “irreducible walks” of length t Theorem: let G be a (d, k) graph of size n. Then for every nontrivial eigenvalue λ: • Then analyze coefficients and structure of Geronimus polynomials to show that if LHS small, eigenvalues must be small.

Conclusion (Part II) • Expanders and degree/diameter graphs two different proposals, from different intuitions • But any good degree/diameter graph is a good expander! • Suggests that good performance of degree/diameter graphs because of good expansion • Expanders easier to construct: just use a good expander • Open questions: • What is true relationship? No reason to think our bounds tight • Moore bound possibly weak: is an optimal degree/diameter graph a good expander even if not close to Moore?

Thank you!