Internet Routers II Using the Pigeon Hole Principle

Internet Routers –II Using the Pigeon Hole Principle to Model Routers Stochastic Networks Seminar March 1 st 2002 Sundar Iyer High Performance Networking Group, Department of Computer Science, Stanford University sundaes@stanford. edu http: //www. stanford. edu/~sundaes

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets. Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 2

What is an Ideal Router? R 1 1 R NR R N N R Total Bandwidth: N 2 R Arriving Packets Interconnect Memory Departing Packets Output Queued Switch v Output Queued (OQ) routers are ideal but not practical ü It minimizes the delay faced by a packet The bandwidth to each output is NR, the total bandwidth is N 2 R The cost and power consumption is prohibitive 3

CIOQ Models Arbiter R 1 1 R 2 R R N N N memories Total Bandwidth: NR Arriving Packets Departing Packets Input Queued Switch v R R R 2 R N N N memories Total Bandwidth: 2 NR Arriving Packets R N memories Departing Packets Combined Input-Output Queued Switch CIOQ switches offer some advantages over OQ switches, but are still not practical ü ü They can give the same delay guarantees as OQ switches They need a switching bandwidth of only 2 NR They have high computational complexity The model does not capture many different architectures 4

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets. Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 5

The Single Buffered Router Model R 1 1 R N N Arriving Packets v Interconnect Memory Interconnect R R Departing Packets Single Buffered Routers buffer packets only once • The interconnects may be – physically separate or merged – one of the interconnects may be a simple pass through • The memory can be – centralized or distributed – one or many – statically or dynamically allotted amongst all ports 6

Why a New Model for Routers? v SB Routers comprise a broader class of routers Ø Ø They replace the CIOQ model They also include other interesting router architectures such as the • Parallel packet switch, Parallel shared memory router, Distributed shared memory router, etc. v With this model we can compare these routers to an ideal router and answer Ø Ø Does a router give me the same delay guarantees as an output queued switch? Can a router give me 100% throughput 7

Parallel Packet Switch Slow Speed Output Queued Routers OQ NR/k 1 NR/k OQ rate, NR 2 rate, NR Departing Packets Arriving Packets NR/k OQ NR/k k 8

Parallel Shared Memory Router Slow Speed Parallel Memories 1 rate, NR 2 k 2 NR NR NR rate, NR Memory Manager Arriving Packets Departing Packets 9

Distributed Shared Memory Router Distributed Line Cards rate, R Arbiter 1 S 1 R S 2 R 1 rate, R 2 2 rate, R N N rate, R BW: S 1 NR BW: S 2 NR N memories Arriving Packets Departing Packets 10

How to Compare Routers? Output Queued Router OQ Switch R R 1 2 R N R 1 1 OQ R N =? Single Any Buffered Router SB Switch 1 Yes? Emulate 2 R 1 No R N N R 11

Using the Pigeonhole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 12

An Abstract Model for a Router A Modified Pigeonhole Principle v Consider the following model for a switch Ø Ø There are P pigeonholes, that can contain an infinite number of pigeons Assume that time is slotted, and in any one time slot T, • at most N pigeons can arrive and at most N can depart • at most one pigeon can enter or leave a specific pigeonhole Ø v When a pigeon arrives, we know the exact time slot at which it will depart For any switch: Ø We need to determine the minimum P, such that all N pigeons can be immediately placed in a pigeonhole when they arrive, and can depart at the right time 13

Solving the Abstract Model v When a pigeon arrives in a time slot Ø Ø Ø v No more than N – 1 other pigeons arrive at that timeslot No more than N other pigeons depart at that timeslot No more than N - 1 other pigeons depart at the same time as this pigeon By the Pigeonhole Principle, Ø P = [(N) + (N-1)] + 1 = 3 N – 1 pigeonholes are sufficient 14

The Constraint Set Technique Upon arrival of a packet, find a memory which is free now and which is free when it will depart 15

What prevents us from finding such free memories? v Physical Constraints, which are limitations imposed by the hardware Ø Ø Ø Memory: (E. g. Parallel Shared Memory Router) Can’t access a memory more than a certain number of times in a time period Bus: (E. g. Parallel Packet Switch) Can’t use the same bus simultaneously for more than a certain number of packets Crossbar: (E. g. Distributed Shared Memory Router) Each port -input and port-output may be busy only once in a timeslot 16

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 17

An Example: Parallel Shared Memory (PSM) Router Number Memories BW per Memory Total BW Emulate? 2 N R 2 NR ? ? DRAM consisting of k memories 2 NR Write Access Time = T Read Access Time = T R R 1 1 2 2 R Number Memories BW per Memory Total BW Emulate? 3 N R 3 NR FIFO 4 N R 4 NR Delay R N N Arbiter Arriving Packets Departing Packets 18

Can a PSM Router Emulate a FIFO OQ Router? Let a cell arrive at input “i” at time “t” and be destined to depart from output port “j” at time “DT” v Such a cell must not be written to memories which v 1. Are used to write the other N-1 arriving cells at t. (Write Constraint Set) 2. Are used to read the N departing cells at t. (Read Constraint Set) 3. Will be used to read the N-1 departing cells at DT. (Future Read Constraint Set) There are three constraint sets v Ø By the pigeonhole principle, 3 N memories at rate R, or a memory bandwidth of 3 NR is sufficient 19

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 20

Distributed Shared Memory Router Arbiter R R 1 1 S 1 R S 2 R R R N No. Mem BW per Mem. Total BW Xbar speed Emulate? N 4 R 4 NR FIFO N 6 R 6 NR Delay N BW: S 1 NR BW: S 2 NR N memories Arriving Packets Departing Packets 21

Can a DSM Router Emulate a FIFO OQ Router? Let a cell arrive at input “i” at time “t” and be destined to depart from output port “j” at time “DT” Ø The cell can be written to any intermediate port “x” such that Ø 1. The edge (i, x) is available at time t. Since, no more than N-1 other cells contend to write at time t, no more than floor[(N-1)/s 1] vertices are unavailable. (Write Constraint Set) 2. The edge (x, j) is available at time DT. Since, no more than N-1 other cells contend to leave at time DT, no more than floor[(N-1)/s 2] vertices are unavailable. (Read Constraint Set) There are two constraint sets Ø • • • By the pigeonhole principle, if suffices that floor[(N-1)/s 1] + floor[(N-1)/s 2] < N. Hence if s 1 =s 2 =2, i. e. s=s 1+s 2=4 is enough. A bandwidth of 4 NR is sufficient 22

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 23

An Abstraction of Delay Guarantees 1 Push In First Out (PIFO) Queue = 4 FIFO queues on a single output v Requirement A 2 = 2 Ø Ø Ø v B 1 = 3 Multiple FIFO Queues need to be maintained An arriving packet joins one of the FIFO queues A server serves these queues with different weights D 2 C 3 B 1 A 1 = 4 C 3 = 2 C 2 = 1 D 2 = 2 A 1 D 1 C 1 = 1 D 1 = 1 C 2 C 1 Abstraction Ø Ø A single queue is maintained An arriving packet can be pushed anywhere into the queue The head of the queue is serviced Note: The relative order of cells of the same “color” does not change 24

What is the Problem with PIFO? v There are two problems • • Problem with PIFO: – The constraint set technique depends on being able to predict the departure time and schedule it. The departure time of a cell is not fixed in PIFO Problem with non PIFO order: – When the memory is shared amongst all outputs, the departure order for the router as a whole is not even PIFO, even though each output queue is a PIFO queue. Lets see what causes the latter problem… 25

What causes the non-PIFO Order? N=4 output ports shown in different colors, each of which is a PIFO queue. d 4 c 4 b 4 a 4 d 3 c 3 b 3 a 3 d 2 c 2 b 2 a 2 Departure Order a 3’ d 4 c 4 b 4 a 4 d 3 c 3 b 3 a 3 d 1 c 1 b 1 a 1 d 2 c 2 b 2 a 2 d 1 c 1 b 1 a 1 Relative order has changed. It is not PIFO! d 4 c 4 b 4 a 3 d 3 c 3 b 3 a 3’ Cannot prevent conflicts since a 3 is already written to memory ü Can prevent conflicts since a 3’ is being written now d 2 c 2 b 2 a 2 d 1 c 1 b 1 a 1 26

Steps in PIFO Scheduling Re-ordering the Output pattern d 5 c 5 b 5 a 5 d 4 c 4 b 4 a 4 d 3 c 3 b 3 a 3 d 2 c 2 b 2 a 2 d 1 c 1 b 1 a 8 a 7 a 6 a 5 d 4 d 3 d 2 d 1 c 4 c 3 c 2 c 1 b 4 b 3 b 2 b 1 a 4 a 3 a 2 a 1 a 3’ a 8 a 7 a 6 a 5 d 4 d 3 d 2 d 1 c 4 c 3 c 2 c 1 b 4 b 3 b 2 b 1 a 4 a 3 a 2 a 1 a 7 a 6 a 5 a 4 d 3 d 2 d 1 c 4 c 3 c 2 c 1 b 4 b 3 b 2 b 1 a 3’ a 2 a 1. . ü Due to future pushes a 3’ may end up with a 3, a 4, a 5 ü Due to future pushes a 3’ may end up with a 2, a 1, a 0, . . ü Can prevent conflicts by taking care of past and future N cells 27

Can a PSM Router Emulate a PIFO OQ Router? A cell which arrives at input “i” at time “t”, destined to depart output port “j” at time “DT” , must not be written to memories which v 1. 2. 3. 4. Are used to write the other N-1 arriving cells at t. (Write Constraint Set) Are used to read the N departing cells at t. (Read Constraint Set) Will be used to read the N-1 departing cells of that output before it (Future Read Constraint Set) Will be used to read the N-1 departing cells of that output after it (Future Read Constraint Set) There are four constraint sets v Ø By the pigeonhole principle, 4 N memories at rate R, or a memory bandwidth of 4 NR is sufficient 28

Using the Pigeon Hole Principle to Model Routers 1. Background 2. The Single Buffered Model for Routers 3. An Abstraction of a Switch Using Pigeons 4. Analysis of First In First Out Switches 5. Analysis of Delay Guarantees in Switches 6. Summary of Results Technique: Constraint Sets Parallel Shared Memory Router Distributed Shared Memory Router Parallel Shared Memory Switch 29

Summary Type Num. Mem. BW of Mem. Total BW Arbiter Emulate? OQ N (N+1)R None Yes Shared Memory 1 2 NR Simple Yes IQ N 2 R 2 NR Moderate No CIOQ 2 N 3 R 6 NR Complex Yes PSM k 4 NR/k 4 NR Simple Yes DSM N 6 R 6 NR Simple Yes PPS Nk 6 R/k 6 NR Simple Yes 30