Scalable Multimodule Switches with Quality of Service Thesis

  • Slides: 37
Download presentation
Scalable Multi-module Switches with Quality of Service Thesis Defense Santosh Krishnan sk@cs. columbia. edu

Scalable Multi-module Switches with Quality of Service Thesis Defense Santosh Krishnan [email protected] columbia. edu May 1, 2006 Advisor: Prof. Henning G. Schulzrinne Co-advisor: Dr. Fabio M. Chiussi

Outline Thesis: Santosh Krishnan • Problem Definition – Motivations, list of contributions • Switching

Outline Thesis: Santosh Krishnan • Problem Definition – Motivations, list of contributions • Switching Model: Components • Related work: Formal methods in switching • Buffered Clos Switches – Concept of functional equivalence • BCS: Throughput and Quality of Service – Single-path BCS: CIOQ, aggregation, pipelining – Multi-path BCS: Parallelization • Conclusions May 1, 2006 2

Problem Definition Thesis: Santosh Krishnan Goals: • How to methodically construct a high-capacity switch?

Problem Definition Thesis: Santosh Krishnan Goals: • How to methodically construct a high-capacity switch? • How to design high-performance algorithms for such switches? Importance: • Physical layer improvements: 10 -G Ethernet, OC-768 • Converged network requiring Qo. S: IPTV, MPLS VPN • Case for modular design: component reuse What exists: • Ad-hoc approach to switch design • No benchmarks, varying performance satisfaction – Non-blocking, 100% throughput, nominal capacity May 1, 2006 3

Contributions Thesis: Santosh Krishnan • Taxonomy of multi-module switches: Buffered Clos Switches • Performance

Contributions Thesis: Santosh Krishnan • Taxonomy of multi-module switches: Buffered Clos Switches • Performance framework: Functional equivalence with ideal switch Mimics circuit-switching rigor Applications Combined I/O Queueing • • Qo. S: Online maximal matching Throughput: Critical matching Strict stability: Maximal matching, SOQF Switched Fair Airport matching Parallelization • • Flow-based PPS: Clos fitting Cell-based PPS: Striping, Equal Dispatch May 1, 2006 Aggregation • • Shadow CIOQ and Decompose Virtual Element Queueing Pipelining • • Striping and Equal Dispatch Concurrent Dispatch: 3 D matching Memory Space Memory • • Combination methods Recursive BCS 4

Switching Model Thesis: Santosh Krishnan CPU PPU PPU Switch Fabric Outputs Inputs PPU Slow

Switching Model Thesis: Santosh Krishnan CPU PPU PPU Switch Fabric Outputs Inputs PPU Slow Path PPU Fast Path • Basic property: Contention • Flows: Guaranteed Qo. S, Best-effort • Ideal Switch: Provide bandwidth trunks, sustain link capacity – Black box for network engineering purposes May 1, 2006 5

Switching Model: Components Thesis: Santosh Krishnan Memory Element Link Scheduling Matching: 2 D Mesh

Switching Model: Components Thesis: Santosh Krishnan Memory Element Link Scheduling Matching: 2 D Mesh Buffers Space Element Conflict-free property Matching complexity Constraints: Memory bandwidth Full-mesh circuitry Monolithic OQ Switch: Ideal IQ Switch q Architecture: Interconnect memory and space elements q Algorithms: Meaningfully emulate the ideal switch for throughput and Qo. S May 1, 2006 6

Background: Clos Networks Outputs Inputs- One circuit Thesis: Santosh Krishnan M Recognize: K •

Background: Clos Networks Outputs Inputs- One circuit Thesis: Santosh Krishnan M Recognize: K • Space-time duality • Fitting: matrix decomposition • Strictly non-blocking: K ≥ 2 M – 1 (Clos theorem) • Re-arrangeable: K ≥ M (Slepian-Duguid) May 1, 2006 Fitting Algorithms Inspiration: Replace selected elements with memory 7

Background: CIOQ Switches Thesis: Santosh Krishnan Pro: • Low memory bandwidth Con: Complexity of

Background: CIOQ Switches Thesis: Santosh Krishnan Pro: • Low memory bandwidth Con: Complexity of matching: • Switch size • Frequency • Reconfiguration rate Queue State 3 0 0 1 7 0 5 May 1, 2006 Configuration • Offline: Templates • Maximum, Maximal, Critical • Heuristics 1 0 0 1 0 What performance results when applied to a changing queue state? 8

Background: CIOQ Switch Results Thesis: Santosh Krishnan Based on combinatorics and stability theory Qo.

Background: CIOQ Switch Results Thesis: Santosh Krishnan Based on combinatorics and stability theory Qo. S Bandwidth Trunks • Birkhoff-Von Neumann decomposition (Chang ‘ 99) • Min/Double decomposition (Towles-Dally ‘ 02) • Low-jitter decomposition (Keslassy ‘ 03) Exact 100% Throughput • Batch maximal matching Throughput • Batch critical-maximum matching (Weller-Hajek ‘ 97) 100% Throughput • i. SLIP for Bernoulli uniform (Mc. Keown ‘ 95) • Maximum weight matching (Mc. Keown ‘ 97) • Maximal matching (Dai-Prabhakar ‘ 00) • Critical-maximum for Bernoulli (Iyer-Mc. Keown ‘ 02) OQ Emulation • LOOFA for work conservation (Krishna ‘ 98) • Exact emulation: stable marriages (Chuang ‘ 99) Auxiliary Results: Envelope matching (Kar ‘ 00), Packet-mode matching (Marsan ‘ 02) May 1, 2006 9

Framework: Buffered Clos Switches Thesis: Santosh Krishnan Parallelize: Pool memory resources PPS Definition: •

Framework: Buffered Clos Switches Thesis: Santosh Krishnan Parallelize: Pool memory resources PPS Definition: • Switch size • Type of elements • Number in first stage • Number in second • Speedup Aggregate: Smaller elements CIOQ-A, G-MSM Pipeline: Lower speed, complexity CIOQ-P, G-MSM q Isomorphism: Non-blocking Clos network q Properties: Multi-stage, fully connected, symmetric, uniform May 1, 2006 10

Framework: Functional Equivalence Characterize relative performance: Functional equivalence f 1: Allocate known rates f

Framework: Functional Equivalence Characterize relative performance: Functional equivalence f 1: Allocate known rates f 2: Relative stability for admissible traffic f 3: Per-output relative stability Thesis: Santosh Krishnan Shape: Bandwidth trunks Literature: 100% throughput Work conserving f 4: Strict relative stability: all pairs f 5: Exact emulation q Emulate an ideal switch: exact, asymptotic q Bandwidth trunks, independent throughput optimization May 1, 2006 11

CIOQ: Bandwidth Trunks Thesis: Santosh Krishnan Shaping plus online matching is sufficient for bandwidth

CIOQ: Bandwidth Trunks Thesis: Santosh Krishnan Shaping plus online matching is sufficient for bandwidth guarantees Online Arbitrary Arrivals Rate Matrix Shape/Batch VOQ Offline BVN Templates Cons: Template Storage Centralized rate processing Weight Scheduler Online: Maximal (s=2) Online: Critical (s=1) Split time into intervals: T = GCD (R) Batch traffic in each interval: Simple counters q Extension of Weller-Hajek maximal matching theorem q Clos analogy: Maximal matching as a strategy for orderly assignments May 1, 2006 12

CIOQ: Admissible Traffic Thesis: Santosh Krishnan Best Throughput Results: • • No speedup: MWM

CIOQ: Admissible Traffic Thesis: Santosh Krishnan Best Throughput Results: • • No speedup: MWM (Mc. Keown et al. ), Speedup 2: Maximal (Dai-Prabhakar) Can a simple maximum size matching suffice for admissible traffic? Red Herring! Queue State Critical matching suffices for asymptotic 100% throughput (f 2) 6 3 0 0 1 7 0 5 0 Augment 6 3 0 1 1 7 2 5 2 MSM Critical Matching Intuition: 2 x 2 Line buckets R 1 May 1, 2006 R 2 C 1 C 2 Max 13

CIOQ: Strict Relative Stability Thesis: Santosh Krishnan q Maximal matching: Keeps under-subscribed outputs stable

CIOQ: Strict Relative Stability Thesis: Santosh Krishnan q Maximal matching: Keeps under-subscribed outputs stable (f 3) (s=2) q Shortest Output-Queue First: (f 4) (s=3) § Output element scheduler: Identical to the one in emulated switch § Intuition: Give preference to less congested pairs at the output § Asymptotic emulation of an ideal switch: long-term fairness May 1, 2006 14

Switched Fair Airport Thesis: Santosh Krishnan Integrate two policies M 1 and M 2:

Switched Fair Airport Thesis: Santosh Krishnan Integrate two policies M 1 and M 2: • M 1: Provides bandwidth trunks given rate reservations • M 2: Optimize throughput independent of above rates Multi-phase Combination Exclusive Combination Speedup Required: M 2 M 1 MWM/Critical BVN Maximal Critical S=1 S=2 S=1 2 3 2 2 S=1 Maximal S=2 Maximal matching is additive to any other policy, hence needs the least speedup May 1, 2006 15

CIOQ-A: Aggregation Thesis: Santosh Krishnan Advantages: Smaller space element Lower arbitration complexity Heterogeneous subports

CIOQ-A: Aggregation Thesis: Santosh Krishnan Advantages: Smaller space element Lower arbitration complexity Heterogeneous subports q Shadow-Decompose: CIOQ emulation (f 5) q VEQ Matching: Less complex, only for admissible traffic (f 2) May 1, 2006 16

CIOQ-P: Pipelining Thesis: Santosh Krishnan q Sequential Dispatch: CIOQ emulation (f 5) q Concurrent

CIOQ-P: Pipelining Thesis: Santosh Krishnan q Sequential Dispatch: CIOQ emulation (f 5) q Concurrent Dispatch: q Limited candidates: stale-state issues q 3 D Maximal Matching for relative stability q Striping: Shadow on envelope basis q Equal Dispatch: q Explicitly equalize load q Separate occupancy counters for each SE Implement arbitrarily complex policies! Advantages: Slower space element Lower arbitration complexity May 1, 2006 17

G-MSM: Combination Thesis: Santosh Krishnan Combination methods: CIOQ-A/P No need for independent analysis Recursion

G-MSM: Combination Thesis: Santosh Krishnan Combination methods: CIOQ-A/P No need for independent analysis Recursion possible May 1, 2006 18

PPS: Architecture Thesis: Santosh Krishnan Core Demux Mux Advantages: Reuse low-capacity core switch Implement

PPS: Architecture Thesis: Santosh Krishnan Core Demux Mux Advantages: Reuse low-capacity core switch Implement arbitrarily slow memories! provided Memoryless first and third stages Performance: Emulates OQ switch q Pool the resources on several switching paths q Dual of a CIOQ-P switch q Matching algorithm replaced by load balancing q Sequence control might be necessary May 1, 2006 19

PPS: Flow-based Thesis: Santosh Krishnan Model for clustered routers: q Per-flow path assignment: explicit

PPS: Flow-based Thesis: Santosh Krishnan Model for clustered routers: q Per-flow path assignment: explicit or hashed q No need for sequence control § Memory in first stage § High speedup (Clos fitting) Ø Unbalanced load assignment § Requires knowledge of loads Split flows May 1, 2006 20

PPS: Cell-based Thesis: Santosh Krishnan Uniformly distribute the load of each flow q Premise:

PPS: Cell-based Thesis: Santosh Krishnan Uniformly distribute the load of each flow q Premise: Each core element receives 1/K cells of each flow Striping Equal Dispatch First-stage buffers NK cells Third-stage buffers 0 NK cells (w/ backpressure) Infinite latency Sequence control Issues q Equal dispatch and striping suffice for asymptotic OQ emulation q Bandwidth trunks: Large buffers required May 1, 2006 21

Summary: A Recipe Book Thesis: Santosh Krishnan • Taxonomy of multi-module switches: Buffered Clos

Summary: A Recipe Book Thesis: Santosh Krishnan • Taxonomy of multi-module switches: Buffered Clos Switches • Performance framework: Functional equivalence with ideal switch Applications Combined I/O Queueing • • Qo. S: Online maximal matching Throughput: Critical matching Strict stability: Maximal matching, SOQF Switched Fair Airport matching Parallelization • • Flow-based PPS: Clos fitting Cell-based PPS: Striping, Equal Dispatch May 1, 2006 Aggregation • • Shadow and Decompose Virtual Element Queueing Pipelining • • Striping and Equal Dispatch Concurrent Dispatch: 3 D matching Memory Space Memory • • Combination methods Recursive BCS 22

Avenues for Follow-on Research Thesis: Santosh Krishnan • Efficient policies for multicast • Similar

Avenues for Follow-on Research Thesis: Santosh Krishnan • Efficient policies for multicast • Similar treatment on other interconnection networks • Theory of backpressure: – Recent interest in buffered crossbars • Quality of stability: Average delay analysis • Short-timescale equivalence • Emulation of a finite-memory ideal switch – Interplay of buffer management with matching algorithms May 1, 2006 23

Supporting Slides May 1, 2006

Supporting Slides May 1, 2006

Relevant Publications Thesis: Santosh Krishnan • • Dynamic Partitioning: Switch Memory Management, Infocom ’

Relevant Publications Thesis: Santosh Krishnan • • Dynamic Partitioning: Switch Memory Management, Infocom ’ 99 Packet Switches with Qo. S Support, Hot Interconnects ’ 00 Feedback Control for Distributed Scheduling, Globecomm ’ 00 Buffered Clos Switches, Columbia TR ’ 02 • • • Inverse Multiplexing for Switches, Globecom ’ 98 Switched Connections Inverse Multiplexing, Intl. Conf. ATM ’ 99 Recognition of Parallel Packet Switches, GBN, Infocom ’ 01 Stability Analysis of Parallel Packet Switches, ICC ’ 01 Open-loop Schemes for Multi-path Switches, ICC ‘ 03 May 1, 2006 Switching Algorithms Parallel Switches 25

Proposal Conjectures Proposal: six conjectures • • • Thesis: Santosh Krishnan Maximal matching is

Proposal Conjectures Proposal: six conjectures • • • Thesis: Santosh Krishnan Maximal matching is sufficient to isolate oversubscribed outputs: DONE SOQF is sufficient for strict relative stability: DONE Equal dispatch for strict stability in CIOQ-P: DONE Equal dispatch plus decomposition for strict stability in G-MSM: DONE Rate shaping plus maximal matching suffices for Qo. S in CIOQ: DONE SOQF suffices for long-term fairness in CIOQ: DONE Plus many more to round out the work May 1, 2006 26

Additional Contributions Thesis: Santosh Krishnan Background: Survey of formal methods in switching– a new

Additional Contributions Thesis: Santosh Krishnan Background: Survey of formal methods in switching– a new perspective Applications Combined I/O Queueing • • • Maximal Matching: Delay analysis Perfect Sequences: Uniform Traffic Multicast support using Recycling Aggregation • • Batch Decomposition (Optical) Support for Heterogeneous Subports Pipelining Parallelization • • • Concurrent Dispatch: BVN and SPS SMM Switches: PPS without backpressure Fractional Dispatch for memoryless inputs May 1, 2006 27

Matching Flavors Thesis: Santosh Krishnan Queue State • Maximal matching: Non-idling, greedy 6 3

Matching Flavors Thesis: Santosh Krishnan Queue State • Maximal matching: Non-idling, greedy 6 3 0 0 1 7 0 5 0 Invariant: Non-empty At least one connection in the marked lines • Maximum-size matching: Maximum flow in a bipartite graph – Ford-Fulkerson, Hopcroft-Karp May 1, 2006 28

Matching Flavors (continued) Thesis: Santosh Krishnan • Critical Matching: Covers all critical rows and

Matching Flavors (continued) Thesis: Santosh Krishnan • Critical Matching: Covers all critical rows and columns – Critical line: A line with the maximum sum • Perfect Matching: Each configuration is a permutation • Maximum Weight Matching: Use queue length as weights – Optimization problem: simplex method • Template Matchings: – BVN: Decompose rate matrix as convex combination of permutations – Double: Lower number of permutations, wasted slots – Min: N permutations will cover all entries, large number of wasted slots • Stable Matching: Gale-Shapely algorithm May 1, 2006 29

Stability Theory Thesis: Santosh Krishnan • Lyapunov functions: Kumar-Meyn ‘ 95 – Mechanism to

Stability Theory Thesis: Santosh Krishnan • Lyapunov functions: Kumar-Meyn ‘ 95 – Mechanism to extend Foster’s criterion to a system of queues – Weighted cartesian product of queue lengths – Symmetric and co-positive • Fluid limits: Dai-Prabhakar ‘ 00 – Function of discrete time: Interpolate – Limit: Scale time to infinity F(t) = lim 1/r f(rt) r ∞ – The scaling parameter may be drawn from an increasing sequence rn May 1, 2006 30

CIOQ: Bandwidth Trunks Thesis: Santosh Krishnan Bandwidth Trunk: Timescale = 1/GCD(R) Arrivals into GQ:

CIOQ: Bandwidth Trunks Thesis: Santosh Krishnan Bandwidth Trunk: Timescale = 1/GCD(R) Arrivals into GQ: Bounded admissible Covers all entries in GQ before next batch q Delay comparable to BVN rate decomposition May 1, 2006 31

CIOQ: Perfect Sequences Thesis: Santosh Krishnan • Sub-maximal Perfect Sequence: – A sequence of

CIOQ: Perfect Sequences Thesis: Santosh Krishnan • Sub-maximal Perfect Sequence: – A sequence of N permutations that covers the unit matrix – A repeating sequence guarantees 1/N to each pair – Suffices for 100% throughput to uniform traffic • Simplementation: Staggered round-robin – Not even maximal! Basis for i. SLIP Basis for Atlanta arbitration May 1, 2006 Concurrent SPS for CIOQ-P: K turns in KN slots 32

Hierarchical Scheduling Thesis: Santosh Krishnan May 1, 2006 33

Hierarchical Scheduling Thesis: Santosh Krishnan May 1, 2006 33

CIOQ-P: Equal Dispatch Thesis: Santosh Krishnan Explicitly equalize the load for each input-output pair

CIOQ-P: Equal Dispatch Thesis: Santosh Krishnan Explicitly equalize the load for each input-output pair Implemented as counters No mis-sequencing issues May 1, 2006 34

CIOQ-P: 3 D Maximal Matching Thesis: Santosh Krishnan Concurrent traversal of queue state matrix

CIOQ-P: 3 D Maximal Matching Thesis: Santosh Krishnan Concurrent traversal of queue state matrix Pointers do not coincide with each other May 1, 2006 35

Recursive G-MSM SPS Memory element of a G-MSM: Replace with a CIOQ switch May

Recursive G-MSM SPS Memory element of a G-MSM: Replace with a CIOQ switch May 1, 2006 Any matching SPS Thesis: Santosh Krishnan Virtual Element Queues Organized per space element 36

PPS: Data Path Thesis: Santosh Krishnan May 1, 2006 37

PPS: Data Path Thesis: Santosh Krishnan May 1, 2006 37