Infiniband architecture Specification Infiniband architecture specification release 1

• Infiniband architecture – Specification (Infiniband architecture specification release 1. 2, Oct. 5, 2004) available at Infiniband Trade Association (http: //www. infinibandta. org) • Potential improvements

• Infiniband architecture overview

• Infiniband architecture overview – Components: • • Links Channel adaptors Switches Routers – The specification allows Infiniband wide area network, but mostly adopted as a system/storage area network. – Topology: • Irregular • Regular: Fat tree – Link speed: • 2. 5 Gbps (X), 10 Gbps (4 X), and 30 Gbps (12 X).

• Layers: somewhat similar to TCP/IP – Physical layer – Link layer • • Error detection (CRC checksum) flow control (credit based) switching, virtual lanes (VL), forwarding table computed by subnet manager – Not adaptive – Network layer: across subnets. • No use for the cluster environment – Transport layer • Reliable/unreliable, connection/datagram – Verbs: interface between adaptors and OS/Users

• Packet format: • Local Route Header (LRH): 8 bytes. Used for local routing by switches within a IBA subnet • Global Route Header (GRH): 40 Bytes. Used for routing between subnets • Base Transport header (BTH): 12 Bytes, for IBA transport • Reliable datagram extended transport header (RDETH): 4 bytes, just for reliable datagram • Datagram extended transport header (DETH): 8 bytes • RDMA extended transport header (RETH): 16 bytes • Atomic, ACK, Atomic ACK, • Immediate DATA extended transport header: 4 bytes, optimized for small packets. • Invalidate • Invariant CRC and variant CRC: – CRC for fields not changed and changed.

• Local Route Header: – Switching based on the destination port address (LID) – Multipath switching by allocating multiple LIDs to one port

• Local Route Header: – Switching based on the destination port address (LID) – Multipath switching by allocating multiple LIDs to one port • GRH: same format as IPV 6 address (16 bytes address)

• Base transport header:

• Verbs – OS/Users access the adaptor through verbs – Communication mechanism: Queue Pair (QP) • Support the four types of services, including reliable connection service • Each connection takes one QP on each end. • Each QP has a send queue and a receive queue. • Users can post send requests to the send queue and receive requests to the receive queue. • Three types of send operations: SEND, RDMA(WRITE, READ, ATOMIC), MEMORY-BINDING • One receive operation (matching SEND)

• Queue Pair: – The status of the result of an operation (send/receive) is stored in the complete queue. – Send/receive queues can bind to different complete queues. • Related system level verbs: – Open QP, create complete queue, Open HCA, open protection domain, register memory, allocate memory window, etc • User level verbs: – post send/receive request, poll for completion.

• To communicate: – Make system calls to setup everything (open QP, bind QP to port, bind complete queues, connect local QP to remote QP, register memory, etc). – Post send/receive requests. – Check completion. – What if a packet arrives before a receive request is posted? • Not specified in the standard • The right response should be a ‘receiver not ready (RNR)’ error. The sender is back-pressed in this case.

• Infiniband has a perfect software interface (Chien'94 paper): – The network subsystem realizes all user level functionality. – User level accesses to the network interface. A few machine instructions will accomplish the transmission task without involving the OS. – Network supports in-order delivery and fault tolerance. – Buffer management is pushed out to the user.

• Silver. Storm 9024: – – – 24 ports 4 X(10 Gbps) or 8 ports 12 X(30 Gbps) switch type: cut-through switch latency: < 140 ns switch bandwidth: 480 Gbps forwarding table size: 48 K VL support: 8 + 1 management

• Silver. Storm 9240: – 24 expansion slots, each expansion model 12 port 4 X or 4 port 12 X (24 x 12 = 288, 288 by 288 switch) – switch type cut-through – switch latency: < 140 ns to < 420 ns – switch bandwidth: 5. 76 Tbps – forwarding table size: 48 K – VL support: 8 + 1 management

• Potential improvements on Infiniband using compiled communication – Improving the internal Infiniband fabric: • Offline routing for static pattern (static SM for a reduced traffic pattern) can be beneficial for irregular networks. • Simplify the layer architecture by having a direct link model (for known patterns), the header can be simplified, may not matter much (Infiniband layers are thin). • Simplify the protection mechanism. • Circuit switch type Infiniband. • Reliable communication protocol is still needed. • Potential benefits can be evaluated by simulation.

• Improving the messaging software (software to hardware interface): no chance. • Improving the MPI implementation over Infiniband: similar to our current work on Ethernet – Message scheduling for collective/point-to-point communications based on the network topology. – Exploring NIC features (buffers in NIC, multicast) – Reducing the number of instructions in a library routine makes sense. Compiled communication can be used to optimize the MPI library. – Compiled communication can help improving the library implementation (e. g. reducing the number of message copies, early requests posting , using RDMA, etc).

• One particular project: – Design algorithms for Infiniband subnet manager – Improving routing performance for Infiniband subnet manager (SM). • Objective: minimize the maximum channel load for an given traffic pattern • Optimize according to a given pattern: the traffic pattern in an application is usually not all-to-all – Default routing used in IBA SM • For a sparse traffic pattern, the maximum channel load can usually be minimized using the minimim interference principle. – Need to extend minimum interference routing for load balance deadlock free routing. – The best way to realize IBA SM is still not clear (unknown) at this time, we can probably do something here. • Irregular network or Fat tree network