Storage area network and System area network SAN

• Storage area network and System area network (SAN) – What are they? – Network requirements – Hardware/software issues – References: • Ulf Troppens, Rainer Erkens, a nd Wolfgang Muller, “Storage Networks Explained - basic and application of Fibre Channel SAN, NAS, i. SCSI and Infiniband”, John Wiley & Sons, 2004. • W. J. Dally and B. Towles, “Principles and Practices of Interconnection Networks”, Morgan Kaufmann, 2004. • Ajay V. Bhatt, “Creating a Third Generation I/O Interconnect, ” available at http: //www. express-lane. org

• Storage area network (SAN): – Server-centric IT architecture: storage devices exist only with servers

• Storage-centric IT architecture: SCSI cables are replaced by a network (storage is now independent of servers).

• Storage area network (SAN) requirement: – Serial transmission for high speed and long distance – Low transmission errors – Low delay of transmitted data • Needs to make it feel like using a local disk • Low delay is a relative term: – The disk subsystem has around 1 ms – 10 ms latency itself. – The communication protocol should not use CPU.

• Current Storage area network (SAN) technology (IBM): – Fibre Channel – TCP/IP + Gigabit Ethernet (i. SCSI) – Infini. Band

• System area network: a network with a high bandwidth and a low lantency that serves as a connection between computers in a distributed computer system.

• Why system area network: – Historically, the system area network comes with a particular parallel machine (supercomputer, e. g. Cray T 3 D, Cray T 3 E, SGI origin 2000, IBM SP, Thinking machine CM 5, Intel Polygon) • The network is very expensive due to low volume • CPU is two generations behind – A more cost effective way to build these system is to decouple the processor technology from the networking technology. – To form cheaper clusters of workstations with the offthe-shelf system area network technology (compared to traditional supercomputers). – current system area networks: • Myrinet, Quadrics, Infiniband

• System area network requirement: – Low latency and high bandwidth at the application level. • Not just at the hardware level • Not just at the system level – Implicitation: • Hardware, network interface, software messaging layer should work together to achieve the goal. – Infiniband is designed as both storage area network and system area network.

• Hardware issues: – High speed links: • Infiniband: 2. 5 Gbps = 250 MBps, 10 Gbps=1 GBps, 30 Gbps = 1 GBps • Fibre channel: 100 MBps, 200 MBps, 400 MBps, 1 GBps. • Myrinet: up to 9. 6 Gbps • As a reference PCI bus: 100 MBps – NIC may need to attach to the memory bridge

• A typical PC:

• A workstation connected to a system area network:

• When the number of end points is large, multiple switches will be needed. • Topology • Switching • Routing

• Topology – Static arrangement of channels and nodes in an interconnection network – Trade-off between cost and performance • Cost: the number and complexity of chips, density and length of the interconnections, etc. • Performance: – Bandwidth and latency: also depend on other factors other than topology – Topology performance metrics: Bisection bandwidth, diameter, nodal degree, channel load

• A cut of a network is the set of channels that partitions the set of all nodes into two disjoint sets. • A bisection of a network is a cut that partitions the network nodes in roughly half. • The bisection bandwidth of a network is the minimum bandwidth over all bisections of the network. • The diameter of a network is the largest minimal hop count over all pairs of nodes. • Under a particular traffic pattern, the channel that carries the largest fraction of traffic determines the maximum channel load of the topology.

• Example topologies: – Regular or irregular – Regular topologies are mostly derived from two main families: butterflies (k-ary n-flies) or tori (k-ary n-cubes)

• Switching: how a packet pass a switch – Message/packet/flit

• Traditional scheme: store-and-forward – Time = H (S + P/B)

• Cut-through switch: – Forward to the next link after the header flit is received. Stop only when the next hop buffer is not available. – Time = H S + P/B, when S << P/B, the time does not depend on the number of hops!!!

• Wormhole routing: – Cut-through switches still allocate buffer to packets. May require a large amount of buffers – Wormhole routing only allocates buffer for one flit for each packet. – Latency is the same as cut-through switching. – When the packet is block, the whole flit “train” is block, occupying links. • Solution: add more virtual channels.

• The deadlock problem in wormhole routing: – Need deadlock free routing scheme to select the right path

• Cut-through switch and wormhole switch are widely used in system are networks – Routing in such systems is an issue!! – Shortest path routing may result in deadlock. – Deadlock free routing:

• Cut-through switch and wormhole switch are widely used in system are networks – Routing in such systems is an issue!! – Shortest path routing may result in deadlock. – Deadlock free routing: • Basic idea: fix the priority of channels and using the channels with increasing priority. • Example: up/down routing

• Up/down routing: – Select a node as the root – Build a spanning tree from the root – Nodes are partitioned into layers based on the position in the spanning tree – The channel from a lower layer node to a higher layer node is the up link, the channel from a higher layer node to a lower layer node is a down link, channels between nodes in the same layer are marked as up or down link based on the node number – In the valid route: an up channel cannot follow an down channel. – These exists at least one valid path between each pair of nodes.

• Problems with deadlock free routing: – Load balancing is a problem, traffic are not evenly distributed – Non-adaptive version of the deadlock free routing scheme is also a problem • How to map the routes in order to get good performance (metrics: maximum channel load? ) • More on the problem to be discussed later.

• Hardware/software codesign and software API issues: – What functionality should be implemented in the hardware. • E. g. adaptive routing may imply out of order packets – Chien’ 04 paper gives good answers to some of these questions.