Infiniband subnet management Discuss the Infiniband subnet management

Infiniband subnet management • Discuss the Infiniband subnet management system • Discuss fat tree and subnet management in an Infiniband with a fat tree topology. • References • A. Bermudez, R. Casado, F. J. Quiles, T. M. Pinkston, J. Duato, “Evaluation of a Subnet Management Mechanism for Infiniband Networks”, ICPP 2003. • A. Vishnu, A. R. Mamidala, H. Jin, D. K. Panda, “Performance Modeling of Subnet Management on Fat Tree Infiniband Networks using Open. SM”, Workshop on System Management Tools on Large Scale Parallel Systems, Held in Conjunction with IPDPS 2005 • X. Lin, Y. Chung, T. Huang, “A Multiple LID Routing Scheme for Fat-Tree. Based Infiniband Networks. ” IPDPS 2004.

• Infiniband devices and entities related to subnet management • Devices: Channel Adapters (CA), Host Channel Adapters, switches, routers • Subnet manager (SM): discovering, configuring, activating and managing the subnet • A subnet management agent (SMA) in every device generates, responses to control packets (subnet management packets (SMPs)), and configure local components for subnet management • SM exchange control packets with SMA with subnet management interface (SMI).

• Subnet management packets (SMP) – 256 bytes of data – Use unreliable datagram service on the management virtual lane (VL 15) – LID routed: use lookup table forwarding • Use after the subnet is setup. E. g. Check the status of an active port – Direct routed: has the information of the output port for each intermediate hop. • Subnet discovery for the subnet is setup

• Subnet management packets (SMP) – Define the operation to be performed by SM – Get: get the information about CA, switch, port – Set: set the attribute of a port (e. g. LID) – Get. Resp: get response – Trap: inform SM about the state of a local node • A SMA stop sending Trap message until it receives Trap. Repress packet. • Topology information can be obtained by a sweep and by peridical Traps.

• Subnet Management phases: – Topology discovery: sending direct routed SMP to evert port and processing the responses. – Path computation: computing valid paths between each pair of end node – Path distribution phase: configuring the forwarding table

• Subnet discovery – SM starts by sending a direct routed Get SMP to its local node. Upone receiving response, SM sends SMPs with additive depth.

• Path computation: – Compute paths between all pair of nodes – For irregular topology: • Up/Down routing does not work directly – Need information about the incoming interface and the destination and Infiniband only uses destination – Potential solution: » find all possible paths » remove all possible down link following up links in each node » find one output port for each destination – Why is that still working? No clear to me. – Other solutions: destination renaming – Fat tree topology: • What is the best that can be achieved is also not clear.

• Path distribution: – Ordering issue: the network may be in an inconsistent state when partially updated, which may result in deadlock during this period. • Traditional solution, no data packets for a period of time • deadlock free reconfiguration schemes.

• Fat Tree: – A way to build large scale clusters

• Fat Tree: – Routing in a complete fat tree: a up phase and a down phase (always contention free).

• Fat Tree: – The complete fat tree has the scalability problem • The root has a very large nodal degree – How to build a fat tree with nodes that have a constant nodal degree. – M-port n-tree FT(m, n) • m is the number of port per switch • n+1 is the height of the tree • The tree consists of 2*(m/2)^n processing nodes and (2 n-1)*(m/2)^(n-1) switches.

– How is an m-port n-tree FT(m, n) connected? • m is the number of port per switch • n+1 is the height of the tree • The tree consists of 2*(m/2)^n processing nodes and (2 n-1)*(m/2)^(n-1) switches. • A processing node is labeled as P(p_0 … p_{n-1}), – P_0 = 0. . m-1, p_i (i!=0) = 0. . m/2 -1 • A switch is labeled as SW<w_0…w_{n-2}, l> – l = 0. . n-1, – When l=0, w_i = 0. . m/2 -1 – When l!=0, w_0 = 0. . m-1, w_I (I!=0) = 0. . m/2 -1

• Figure 5: a 4 -port 3 -tree

• How is the tree connected? – SW<w, l>_k be the kth port of SW(w, l). – SW(w, l)_k and SW(w’, l’)_k’ is connected iff • l’ = l + 1 • w_0…w_{n-3} = w_0’…w_{l-1}’w_{l+1}’…w_{n-2}’ • k=w_l’, k’ = w_{n-2} + m/2 – Question: which switches are connected to SW<001, l>? How the ports are connected?

• Fat tree properties: – Multiple routes between two nodes • Deterministic routing: one path between two nodes, how to map? – What is a good mapping? • In the case when the traffic pattern is unknown, common practice is to minimize the maximum load on a link. – Do we know how to do it? • Not clear even when there is no restriction on the routing. It is likely that an optimal solution exists for a particular FAT tree topology • In infiniband, destination based routing put some restriction on which path can be used.