Using optics to scale Internet Routers Computer Forum

Using optics to scale Internet Routers Computer Forum, May 2003 Nick Mc. Keown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford. edu www. stanford. edu/~nickm 1

Problems facing routers 1. The problem: Ø Ø Ø 2. Capacity scales slower than user traffic. Power limits capacity. All-optical routers are infeasible. Our approach Ø Ø Explore how optics can be used inside routers to reduce power, and therefore scale capacity. Design a high capacity router that exposes the problems, and leads to interesting research questions. 2

Internet Routers Line Capacity 2 x / 7 months User Traffic 2 x / 12 months Router Capacity 2. 2 x / 18 months Moore’s Law 2 x / 18 months DRAM Random Access Time 1. 1 x / 18 months 3

Power consumption of singlerack Internet routers 4

Multi-rack distributed routers reduce power density Optical links 100 s of metres Switch Core Linecards 5

Motivating Design: 100 Tb/s Optical Router Optical Switch ic n o r ct rd #1 e l E eca n i L 60 - s 1 ion sing Gb/ t a in ces 320 rm o /s b G 0 16 e te et pr ring n i • L pack buffe • IP cket a • P Request 160 320 Gb/ s Ele c Lin tronic eca rd # 625 • Li n • IP e term • Pa packe inatio n t cke t bu proces ffer s ing Arbitration Grant 40 Gb/s (100 Tb/s = 625 * 160 Gb/s) 6

Research Groups Mark Horowitz horowitz@ee. stanford. edu v Nick Mc. Keown nickm@ee. stanford. edu v Olav Solgaard olav@ee. stanford. edu v David Miller dabm@ee. stanford. edu v v 8 -10 Ph. D students 7

100 Tb/s optical router v Objective Ø To determine the best way to incorporate optics into routers. Ø Push technology hard to expose new issues. • Photonics, Electronics, System design Ø Motivating example: The design of a 100 Tb/s Internet router • Challenging but not impossible (~100 x current commercial systems) • It identifies some interesting research problems 8

Research Problems v Linecard Ø v Architecture Ø v Memory bottleneck: Address lookup and packet buffering Arbitration: Computation complexity Switch Fabric Ø Ø Ø Optics: Fabric scalability and speed Optics: Optical modulators Electronics: Low power optical links Electronics: Optical switch control Electronics: Clock recovery for intra-system links Packaging. 9

Outline v Load-Balanced Switch Overview v Passive Mesh Paradigm v WGR-based Switch Fabric v Hybrid Optical-Electrical Switch Fabric 10

The Arbitration Problem A packet switch fabric is reconfigured for every packet transfer. v At 160 Gb/s, a new IP packet can arrive every 2 ns. v The configuration is picked to maximize throughput and not waste capacity. v Known algorithms are too slow. v 11

Load-Balanced Switch External Inputs 1 N Load Balancing Load-balancing cyclic shift v v Internal Inputs External Outputs 1 1 N N Switching cyclic shift First stage load-balances incoming flows Second stage is the usual switching cyclic shift 12

Load-Balanced Switch External Inputs 1 12 Internal Inputs External Outputs 1 1 1 2 N N Load-balancing cyclic shift N Switching cyclic shift 100% throughput for broad range of traffic types (C. S. Chang et al. , 2001) 13

Outline v Load-Balanced Switch Overview v Passive Mesh Paradigm v WGR-based Switch Fabric v Hybrid Optical-Electrical Switch Fabric 14

Passive Mesh 1 2 3 R R Cyclic Shift 1 2 3 R/N Passive mesh 1 2 3 2 R/N 1 2 3 Passive mesh No more arbitrations, no more reconfigurations! 15

Outline v Load-Balanced Switch Overview v Passive Mesh Paradigm v WGR-based Switch Fabric v Hybrid Optical-Electrical Switch Fabric 16

AWGR (Arrayed Waveguide Grating Router) A Passive Optical Component 1 1 1 Linecard 1 l 1, l 2 …l N l 11 Linecard 2 l 12 Linecard 2 l 1 N Linecard N Nx. N WGR Linecard N v v Wavelength i on input port j goes to output port (i+j-1) mod N Can shuffle information from different inputs 17

WGR Based Solution Fixed Laser/Modulator l 1 Linecard 1 l 2 l. N l 1 Linecard N l 2 l. N 1 1 l 1, l 2 1 …l N l 2 1 , l 2 2 2 …l N N N l 1, l 2 N …l N Detector 1 N l 1, l 2 2 …l N 2 l 1, l 3 Nx. N WGR … l N N 1 2 l 1 l 2 Linecard 1 l. N l 1 l 2 Linecard 2 l. N N-1 l 1, l 2 1 …l N l 1 l 2 Linecard N l. N 18

Switch fabric design v Design a switch fabric Ø Ø v For load-balancing and switching stages 625 ports of 2 x 160 Gbps Features: Ø Ø Flexibility: arbitrary addition and deletion of linecards (due to upgrades/failures) Scalability 19

Outline v Load-Balanced Switch Overview v Passive Mesh Paradigm v WGR-based Switch Fabric v Hybrid Optical-Electrical Switch Fabric 20

From Linecard Mesh to Group Mesh Linecard 1 2 R/6 Linecard 1 Linecard 2 Linecard 3 Linecard 4 Linecard 5 Linecard 6 21

From Linecard Mesh to Group Mesh Linecard 1 Linecard 2 Linecard 1 3 R Group 1 Linecard 3 3 R Linecard 4 Linecard 5 Linecard 6 Linecard 2 Group 2 3 R 3 R Linecard 4 Group 2 Linecard 5 Linecard 6 22

Example 2 R/3 Linecard 1 Linecard 2 Linecard 3 23

Example 8 R/3 Linecard 1 Crossbar Linecard 2 4 R/3 Linecard 3 Crossbar 4 R/3 2 R/3 Linecard 3 Crossbar 24

Example Static MEMS 4 R/3 Linecard 1 Linecard 2 Linecard 3 4 R/3 2 x 3 Crossbar 2 R/3 2 x 3 Crossbar 4 R/3 2 x 3 Crossbar Linecard 1 Linecard 2 Linecard 3 25

Hybrid Switch Fabric Electronic Switches 1 Linecard 2 Static MEMS Fixed Lasers 2 Lx. M Crossbar 3 1 2 Lx. M Crossbar Linecard 2 Lx. M Crossbar Group G Linecard 2 Linecard L Group 2 2 M Mx. L Crossbar M 2 Gx. G MEMS Linecard 1 2 3 Gx. G MEMS Linecard 2 Linecard L 1 1 M Linecard L 2 1 3 Mx. L Crossbar Group 1 3 Group 2 Linecard 1 3 M Linecard L Linecard 1 2 Gx. G MEMS Group 1 Linecard 1 1 1 M Linecard L Linecard 2 Gx. G MEMS Optical Electronic Receivers Switches 3 M Linecard 1 Mx. L Crossbar Linecard 2 Linecard L Group G 26

Conclusion Power: 100 Tb/s optical switch fabric consumes almost no power. v Optics: No optical components reconfigured on packet-by-packet basis. v Capacity: No centralized arbitration and scheduling algorithms. v Throughput: 100% throughput guarantee. v 27