Carnegie Mellon ISIS Next Generation Router Shahzad Ali

Carnegie Mellon ISIS Next Generation Router Shahzad Ali Xia Chen Brendan Howell Yu Zhong Broadband Networks, 2000

Motivation Carnegie Mellon l Anybody can make a router. �Key is to make one with – High speed (OC-48 and up) – High port densities – Better performance (throughput, delay, latency) – Cheap – etc. l But we always forget some requirements. �The new network requires new services – Guaranteed bandwidth/latency (Qo. S) – Scaleable Design Now the task is not that simple!! Broadband Networks, 2000

Possible uses for our router Carnegie Mellon l l l l l Aggregation point for large data centers Backbone router at a major peering point Backbone router for carrier facilities Core router for IP transit providers High bandwidth Scaleable Qo. S Robustness Major Routing Protocol support. Broadband Networks, 2000

So what did we come up with? Carnegie Mellon l l l l Design a router that meets the minimum specifications Maximize switch throughput Provide support for strong Qo. S in the switch with realistic assumptions Build an extensible simulator Evaluate the design using simulations Make the design realistic! Tackle the tough issue of scalability Result: ISIS Broadband Networks, 2000

Design Decisions Carnegie Mellon l 64 port OC-48 with total capacity of 160 Gb/s �Our base design allows for 128 OC-48 (320 Gb/s) �Uses PMC Sierra PM 9311 and PM 9312 crossbar and scheduler. l Buffers �PC-100 10 ns access latency �Dimension buffer by simulation l Queuing? �Output Queuing �Input Queuing Broadband Networks, 2000

Output Queuing Carnegie Mellon l Provides perfect Qo. S. l Switching Fabric � Can order outgoing cells according to their priorities using FQ (WFQ, WF 2 Q+, etc) � Use fast sorting techniques to speedup the process*. But, requires a speedup of N for switching fabric � Not scaleable for high bandwidth switches. � Not cheap either. Input *“A simple and Fast Hardware Implementation Architecture for WFQ Algorithms”, Nen-Fu Huang and Chi-An Su Broadband Networks, 2000 Output

Input Queuing Carnegie Mellon l Provides no support for hard guarantees (Qo. S). l Switching Fabric � Difficult to maximize switch throughput while providing these guarantees. But, simple to implement, with minimal speedup and maximum throughput. � Make scheduling decisions using a smart scheduling algorithm. � Use i. Slip or similar algorithms to achieve high throughput Broadband Networks, 2000 Input Output

Queuing Carnegie Mellon l Output Queuing �No! �Violates our design goal of scalability. �Too Expensive l Input Queuing �Maybe �If we can do proper Qo. S with it. l Maybe a third option? Broadband Networks, 2000

Qo. S with Input Queuing Carnegie Mellon l l l l The only way we can imagine this happening is with buffered crossbars The buffers are provided at each cross-point Number of buffers are bounded (between 3 -5 cells) Use FQ servers at Input, crossbar buffers and output to achieve Qo. S The solution is hard to implement Plus it only provides probabilistic guarantees Plus we don’t even know if it actually works Broadband Networks, 2000

Qo. S with Input Queuing Carnegie Mellon l Some recent work focuses on this aspect. �“Implementing Distributed Packet fair Queuing in a Scalable Switch Architecture”, D. C. Stephens �“A Distributed Scheduling Architecture for Scalable Packet Switches”, F. M. Chiussi and A. Francini Is that the best we can do? Broadband Networks, 2000

CIOQ Carnegie Mellon l Combined Input output queuing: combines the benefits of both Input and Output queuing. �Input queuing is speedup of 1 �Output queuing is speedup of N. �CIOQ is between 1 and N, need to buffer at input and output. l Requires a speedup of only 2 to simulate an output queuing switch completely. �Reasonable given the difficulty of our initial goal. l Guarantees the exact same output as a OQ switch. Broadband Networks, 2000

CIOQ Carnegie Mellon l Assigns a value called slackness to each VOQ at the input. �Slackness is the urgency of a packet. Slackness of 0 means high urgency. �Inputs and outputs select ports based on a priority calculated by the FQ discipline used. – For FCFS, high priority for packet that came earlier. l l Uses Gale-Shapely algorithm to do a stable matching of inputs to outputs after they have been selected in Phase I. No analysis has been done on the throughput of a CIOQ switch, just an analytical bound on its proximity to OQ switch. Broadband Networks, 2000

CIOQ Carnegie Mellon l Some recent work in this area �“Delay-Bound Guarantee in CIOQ Switches”, H. Chao, L. Shen Chen �“Matching Output Queuing with CIOQ Switch”, S. Chaung, A. Goel, N. Mc. Keown, B. Prabhakar. �“College Admissions and the stability of marriage”, D. Gale, L. Shapely. Broadband Networks, 2000

ngr. Sim Carnegie Mellon l l A simulator for ISIS: Next Generation Router. ngr. Sim follows a modular design approach. �All components are coded as modules that can be plugged in place of others. �Allows for easy mix-and-match of various schemes. l It is event driven … completely. �Everything is driven by the event queue. Broadband Networks, 2000

Event handling Carnegie Mellon l l An abstract class handler is defined. A class event is described which consists of an object of type handler. � The handler is supposed to be the component that is responsible for the event. � When the event is run, the handle function of the handler is called. l Event are enqueued into an event queue. class handler { public: handler() { }; virtual ~handler() { }; virtual void handle(event *e) = 0; void set_next_handler(handler * h) { next_handler = h; } protected: handler * next_handler; }; Event Port ID Data Handler Broadband Networks, 2000

Event handling Carnegie Mellon l Efficient event handling capability. �Events are queued in a structure called a Skip-list, developed by William Pugh at University of Maryland*. �A probabilistic list which allows for O(1) inserts, deletes and searches. l l Simulation time is kept track of in the event queue. The event at the head of the event queue is run (handle function called) and the simulation time is updated. *http: //www. cs. umd. edu/~pugh Broadband Networks, 2000

Main Loop for ngr. Sim Carnegie Mellon // Initialize the components tgen, ipp, // framer, sched, fab, reframer, opp ((crossbar*)fab)->start(); ((tgen*)tg)->start(); while (1){ event* e = (eq. instance()). dequeue(); if (e == NULL && !sim_running) break; (e->get_next_handler())->handle(e); if ((eq. instance()). get_time() > sim. Time){ sim_running = 0; ((tgen*)tg)->stop(); ((crossbar*)fab)->stop(); } } // print Stats Broadband Networks, 2000 Initialize the switch and declare various components. ALL READ FROM A CONFIGURATION FILE Start the periodic fabric pull and the traffic generation events. Main Loop Get the event of the head of the queue. This updates the system time as well Call the handle function. Check if simulation ended Print the statistics and draw graphs

Design of ngr. Sim Carnegie Mellon l The design follows exactly from the design we had in the earlier design review. Broadband Networks, 2000

Object Diagram for ngr. Sim Carnegie Mellon (Variable Length) (64 Bytes, 6 Bytes Fixed Header) Packet TGEN FRAMER IPP Route Lookup Checksum Break packet into cells Etc Scheduler Traffic Cell Fabric Traffic FIFO i. Slip Put packets on the outgoing link CIOQ Use FQ here as well OPP REFRAMER Assemble cells back to packets simple. OPP Broadband Networks, 2000 cioq. OPP Crossbar

Description of the components Carnegie Mellon l l Packets are generated in TGEN module and are passed as events to IPP, Framer and Scheduler. Framer breaks packets into cells. �Cells are 64 bytes with 6 bytes of header. – 2 input + output port number = 4 bytes – 1 byte for cell ID – 1 byte for flag + priority l Scheduler is an abstract class. All schedulers that are implemented derive from this base class. �We have FCFS, i. Slip and CIOQ implemented. l The scheduler does not pass the packets on to the fabric. The fabric is running at a fixed time slot which is determined by the cell size and line speed. Broadband Networks, 2000

Description of the components Carnegie Mellon l l The fabric pulls cells from the scheduler at these regular intervals. Fabric is also an abstract class. All fabrics will derive form this class. �We currently have a crossbar fabric implemented. l The fabric sends the cells to the reframer. �Reframer reassemble cells into packets. �If some cell is not received in a certain time limit, the partial packet is discarded. l The reframer passes the reassembled packets to the OPP. �The OPP queues the packets and sends them out at the link rate. �We have simple. OPP and cioq. OPP implemented. Broadband Networks, 2000

Statistics Carnegie Mellon l All modules have statistics. �They calculate them independently and can be queried. �We keep track of – cell and packet count – buffer sizes – delays – drops l We ran experiments for 100 million clock ticks. �Each data point is a result of 10 runs of the same experiment with different random seeds. �The values were only collected after steady-state. �Due to time-constraint, we could only run for 16 ports. Broadband Networks, 2000

Carnegie Mellon Throughput (FIFO) Theoretical bound = 58 % Scheduler throughput is 45% ? Overall throughput is 30% Broadband Networks, 2000

Carnegie Mellon Throughput (speedup = 1. 25) Theoretical bound = 58 % Scheduler throughput is 55% Overall throughput is 40 % Broadband Networks, 2000

Carnegie Mellon Broadband Networks, 2000 FIFO with different Speedup

Carnegie Mellon Broadband Networks, 2000 FIFO with different Speedup

Carnegie Mellon Scheduling Algorithm Throughput Broadband Networks, 2000

Carnegie Mellon Scheduling Algorithm Delay ? Broadband Networks, 2000

Carnegie Mellon Broadband Networks, 2000 Buffer sizes with i. Slip

Carnegie Mellon Scheduling Algorithm i. Slip performs worse than CIOQ Broadband Networks, 2000

To make ISIS a reality … Carnegie Mellon l Physical Specifications of base design �Chassis Height: 10 in. Includes power module shelf (AC or DC) �Chassis Width: 17. 25 in. not including rack mount flanges. can be rack mounted in 19 or 22 in. �Chassis Depth: 18 in. not including cable management system �Chassis Weight: approximately 50 lbs. depending on configuration l Standards compliance �Safety: UL 1950, IEC 60825, TS 001, AS/NZS 3260 �Electromagnetic Emissions: FCC Class A, ICES-003 Class A, EN 55022 Class B, VCCI Class B, AS/NZS 3548 Class B �NEBS: SR-3580 Level 3 Compliant Broadband Networks, 2000

Chassis Configuration Carnegie Mellon l l Modular Design separates Line card module from Switch fabric module. Redundant Power supplies and fabrics can be used to ensure robustness. Compact design allows flexibility in rack placement. The modules are connected through proprietary fiber interconnect using LCS protocol*. *http: //www. pmcsierra. com/products/details/pm 9311/ Broadband Networks, 2000

That’s cute … but show me something big! Carnegie Mellon l l The PMC 9113 has 10 Gb/s channels that can be used to connect to other switching modules or line cards through fiber and LCS protocol. So instead of using one 320 Gb/s switch, we can use more than one. We can extend the capacity of the switch by connecting many switch modules in a structure. The easiest structure we can think of is a mesh. �Other possibilities include a hyper-cube. Broadband Networks, 2000

ISIS-A: ISIS with an attitude Carnegie Mellon Routing Module 16 10 Gb/s Channels to other switching modules Switching module with 320 Gb/s capacity Line card Module (160 Gb/s) 10 Gp/s channel 16 such switches connected in a mesh through 10 Gb/s channels of fiber. Broadband Networks, 2000 Full mesh of 16 equals 1024 OC-48 ports (2. 5 Tb/s)

ISIS-A: Routing Carnegie Mellon l Based on source and destination address, line cards route to �ports on the card itself (through the line card) �Other line cards on same module (through the fabric) �Other switching modules (through the interconnect) l Routing between switching modules based on modified hot-potato routing with queue lengths �Queues only monitor bandwidth utilization of channels between modules. – Built into the chip-set. �Send to the recipient directly if queue is not loaded. �Send to the least loaded queue recipient, otherwise. Broadband Networks, 2000

ISIS-A: Routing and Qo. S. Carnegie Mellon l Routing is more like load balancing �Only more intelligent �Controls delays l For Qo. S, �Some bandwidth is reserved for guaranteed traffic (max 10 Gb/s between two modules) �If such traffic arrives, send directly to the recipient (one link) using this reserved bandwidth. �If no such traffic, then use the bandwidth for normal traffic. l Requires modification to the scheduler (which is programmable) �Addressing has to be universal. �Routing between modules has to be added. Broadband Networks, 2000

Future Work Carnegie Mellon l Short-term �Need to run more experiments with different parameters. �Plan to follow-up on the scalability options. �Get statistical significance for the results. l Long-term �Implement other scheduling algorithms. �Implement other fabric types. �Explore other possibilities for Qo. S with IQ switches. Broadband Networks, 2000