Interconnection Networks Nanjing University 2016 Indirect Networks or

Interconnection Networks Nanjing University 2016 Indirect Networks or Dynamic Networks Guihai Chen …with major presentation contribution from José Flich, UPV (and Cell BE EIB slides by Tom Ainsworth, USC)

Questions in mind Difference between Static/direct and Dynamic/indirect Networks Nanjing University 2016 Why Multi-Stage Interconnection Networks Large Switch and Small Switch How to design non-blocking MINs 2

Outline Network Topology Preliminaries and Evolution Centralized Switched (Indirect) Networks Nanjing University 2016 From non-blocking crossbar to blocking MINs From blocking MINs to non-blocking MINs 3

Network Topology Preliminaries and Evolution • One switch suffices to connect a small number of devices – Number of switch ports limited by VLSI technology, power Nanjing University 2016 consumption, packaging, and other such cost constraints • A fabric of interconnected switches (i. e. , switch fabric or network fabric) is needed when the number of devices is much larger – The topology must make a path(s) available for every pair of devices—property of connectedness or full access (What paths? ) • Topology defines the connection structure across all components – Bisection bandwidth: the minimum bandwidth of all links crossing a network split into two roughly equal halves – Full bisection bandwidth: › Network BWBisection = Injection (or Reception) BWBisection= N/2 – Bisection bandwidth mainly affects performance • Topology is constrained primarily by local chip/board pin-outs; secondarily, (if at all) by global bisection bandwidth 4

Network Topology Preliminaries and Evolution Nanjing University 2016 • Several tens of topologies proposed, but less than a dozen used • 1970 s and 1980 s – Topologies were proposed to reduce hop count • 1990 s – Pipelined transmission and switching techniques – Packet latency became decoupled from hop count • 2000 s – Topology still important (especially OCNs, SANs, P 2 P Overlays, DCNs) when N is high – Topology impacts performance and has a major impact on cost 5

Network Topology Centralized Switched (Indirect) Networks • Crossbar network – Crosspoint switch complexity increases quadratically with the Nanjing University 2016 number of crossbar input/output ports, N, i. e. , grows as O(N 2) – Has the property of being non-blocking 0 1 2 3 4 5 6 0 7 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 1 2 3 4 5 6 7 6

Network Topology Nanjing University 2016 From Crossbar to MINs 7

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 • Multistage interconnection networks (MINs) – Crossbar split into several stages consisting of smaller crossbars – Complexity grows as O(N × log N), where N is # of end nodes – Inter-stage connections represented by a set of permutation functions 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 Omega topology, perfect-shuffle exchange 8

Network Topology Appendix Nanjing University 2016 • Shuffle function – N= 0…N-1 – f(i)= 2 i, when I <N/2 – f(i)=(2 i+1), mod N when i ≥ N/2 – Often used as a connection pattern – unshuffle function, also often used 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 perfect-shuffle unshuffle 9

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 16 port, 4 stage Omega network 10

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 16 port, 4 stage Baseline network 11

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 16 port, 4 stage Butterfly network 12

Network Topology-Correction to Butterfly Centralized Switched (Indirect) Networks Nanjing University 2016 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 16 port, 4 stage Butterfly network 13

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 16 port, 4 stage Cube network 14

Network Topology Centralized Switched (Indirect) Networks • Multistage interconnection networks (MINs) – MINs interconnect N input/output ports using k x k switches Nanjing University 2016 › logk. N switch stages, each with N/k switches › N/k(logk. N) total number of switches – Example: Compute the switch and link costs of interconnecting 4096 nodes using a crossbar relative to a MIN, assuming that switch cost grows quadratically with the number of input/output ports (k). Consider the following values of k: › MIN with 2 x 2 switches › MIN with 4 x 4 switches › MIN with 16 x 16 switches 15

Network Topology Centralized Switched (Indirect) Networks • Example: compute the relative switch and link costs, N = 4096 Nanjing University 2016 cost(crossbar)switches = 40962 cost(crossbar)links = 8192 relative_cost(2 × 2)switches = 40962 / (22 × 4096/2 × log 2 4096) = 170 relative_cost(2 × 2)links = 8192 / (4096 × (log 2 4096 + 1)) = 2/13 = 0. 1538 relative_cost(4 × 4)switches = 40962 / (42 × 4096/4 × log 4 4096) = 170 relative_cost(4 × 4)links = 8192 / (4096 × (log 4 4096 + 1)) = 2/7 = 0. 2857 relative_cost(16 × 16)switches = 40962 / (162 × 4096/16 × log 16 4096) = 85 relative_cost(16 × 16)links = 8192 / (4096 × (log 16 4096 + 1)) = 2/4 = 0. 5 16

Network Topology Centralized Switched (Indirect) Networks • Relative switch and link costs for various values of k and N (crossbar relative to a MIN) Nanjing University 2016 Relative switch cost Relative link cost 17

Network Topology Nanjing University 2016 From blocking to non-blocking again 18

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 • Reduction in MIN switch cost comes at the price of performance – Network has the property of being blocking – Contention is more likely to occur on network links › Paths from different sources to different destinations share one or more links 0 1 2 3 4 5 6 7 0 0 1 2 3 4 5 6 7 non-blocking topology X 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 blocking topology 19

Network Topology Centralized Switched (Indirect) Networks • How to reduce blocking in MINs? Provide alternative paths! – Use larger switches (can equate to using more switches) Nanjing University 2016 › Clos network: minimally three stages (non-blocking) » A larger switch in the middle of two other switch stages provides enough alternative paths to avoid all conflicts – Use more switches › Add logk. N - 1 stages, mirroring the original topology » Rearrangeably non-blocking » Allows for non-conflicting paths » Doubles network hop count (distance), d » Centralized control can rearrange established paths › Benes topology: 2(log 2 N) - 1 stages (rearrangeably non-blocking) » Recursively applies the three-stage Clos network concept to the middle-stage set of switches to reduce all switches to 2 x 2 20

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port Crossbar network 21

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port, 3 -stage Clos network 22

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port, 5 -stage Clos network 23

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port, 7 stage Clos network = Benes topology 24

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology 25

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology 26

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology 27

Network Topology Centralized Switched (Indirect) Networks Nanjing University 2016 • • Bidirectional MINs Increase modularity Reduce hop count, d Fat tree network Network Bisection 0 1 2 3 – Nodes at tree leaves – Switches at tree vertices – Total link bandwidth is constant across all tree levels, with full bisection bandwidth – Equivalent to folded Benes topology – Preferred topology in many SANs 4 5 6 7 8 9 10 11 12 13 14 15 Folded Clos = Folded Benes = Fat tree network 28

Network Topology Myrinet-2000 Clos Network for 128 Hosts Nanjing University 2016 • Backplane of the M 3 E 128 Switch • M 3 -SW 16 -8 F fiber line card (8 ports) http: //myri. com 29

Network Topology Myrinet-2000 Clos Network for 128 Hosts Nanjing University 2016 • “Network in a Box” • 16 fiber line cards connected to the M 3 -E 128 Switch backplane http: //myri. com 30

Network Topology Myrinet-2000 Clos Network Extended to 512 Hosts Nanjing University 2016 http: //myri. com 31

Assignment 2 -2 Nanjing University 2016 Chose one of the following exercises: • Calculate how many permutations nxn Omega network could support. • Prove that folded Clos network, folded Bens network, and Fat tree network are isomorphic to each other. Parallel Processing, Low-Diameter Architectures Slide 32 SJTU@Fall 2012