Investigating Serial Attached SCSI SAS over TCP t

Investigating Serial Attached SCSI (SAS) over TCP (t. SAS) UCCS Master’s Project Deepti Reddy

Outline of the Talk § Introduction to t. SAS and related technology § Motivation and Challenges § t. SAS Design § Performance Evaluation of a Mock Application § Lessons Learnt and Future Directions § Conclusion 11/3/2011 t. SAS/sreddy 2

SCSI vs SAS SCSI (Small Computer System Interface) • • The SCSI protocol is an application layer storage protocol. It's a standard for connecting peripherals to your computer via a standard hardware interface, which uses standard SCSI commands. SAS (Serial Attached SCSI) • • • SAS is the successor of SCSI technology Initially introduced at 3 Gb/s in 2004 Currently, supports 6 Gb/s 12 Gbps by 2012 Significantly increased the available bandwidth offered by legacy SCSI storage systems. Use of expanders increases the scalability over legacy SCSI 11/3/2011 t. SAS/sreddy 3

SCSI vs SAS 11/3/2011 SCSI SAS Topology Parallel Bus Serial Bus Reduces Noise (Better signal integrity) Speed 3. 2 Gbps 3 Gbps, 6 Gbps, 12 Gbps Distance 1 to 12 meters 8 meters Number of Targets 14 devices 128 expanders. > 16, 000 with cascaded expanders Devices SCSI only SAS & SATA Used widely in Data Centers today Connectivity Single Port Dual Port Fail-over/fail-back Drive Form Factor 3. 5” 2. 5” Cost Low Medium 4 t. SAS/sreddy

i. SCSI protocol describes a means of transporting of the SCSI packets over TCP/IP § The i. SCSI specification is revised and updated by the Internet Engineering task Force (IEFT). § Work by S. Aiken, D. Grunwald, A. Pleszkun and J. Willeke shows the performance of a commercial i. SCSI software implementation compared quite favorably with fibrechannel [7] i. SCSI protocol layering model 11/3/2011 t. SAS/sreddy 5

Typical SAS Topology SAS CONTROLLER (HOST BUS ADAPTER) Driver ~8 m X X X - DISK DRIVES X - EXPANDERS 11/3/2011 t. SAS/sreddy 6

Motivation & Challenges § Overcome the distance and scalability limitations of traditional Serial Attached SCSI (SAS) by using t. SAS. § Like i. SCSI, t. SAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations § Provide research results for future industry specification for t. SAS and i. SCSI. 11/3/2011 t. SAS/sreddy 7

Related Work 11/3/2011 § Michael Ko’s patent at IBM Research-Almaden on Serial Attached SCSI over Ethernet proposes a very similar solution to the t. SAS solution provided in this project. § i. SCSI specification (SCSI over TCP) itself is similar to a t. SAS solution (SAS over TCP). The i. SCSI solution can be heavily leveraged for a t. SAS solution. § The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a t. SAS solution. t. SAS/sreddy 8

t. SAS Topology t. SAS HBA t. SAS CONTROLLER/HBA SCSI Driver X - TCP link - DISK DRIVES X - SAS link - EXPANDERS 11/3/2011 t. SAS/sreddy 9

t. SAS Topology § The topology for t. SAS on the previous slide (similar to Michael Ko’s proposal) can be extended such that t. SAS is also used between a HBA and an expander in a SAS topology. This helps increase scalability of t. SAS. 11/3/2011 t. SAS/sreddy 10

t. SAS Topology t. SAS HBA t. SAS CONTROLLER/HBA SCSI Driver X - TCP link - DISK DRIVES X - SAS link - EXPANDERS 11/3/2011 t. SAS/sreddy 11

Goals of the project 11/3/2011 § Investigate how t. SAS can be implemented § Design and develop a mock t. SAS application − Write a client program using C/C++ that sends a set of t. SAS commands to a server. − Write a server program using C/C++ that receives a set of t. SAS commands, processes them and responds to the client with t. SAS responses. § Evaluate the t. SAS solution and compare with a mock i. SCSI client/server application. t. SAS/sreddy 12

Software and Hardware solutions for t. SAS implementations 11/3/2011 § Software based t. SAS solution − Cheaper than a hardware based solution − All t. SAS processing is done by the processor and TCP/IP operations are also executed by the CPU − NIC is merely an interface to the network § TCP Offload Engine (TOE) solution − NICs with integrated TOEs have hardware built into the card that allows the TCP/IP processing to be done at the interface § Hardware based t. SAS solution − In a hardware-based t. SAS environment, the initiator and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and t. SAS processing t. SAS/sreddy 13

Software and Hardware solutions for t. SAS implementations § Broadcom on their h/w i. SCSI solutions states the following “Software i. SCSI initiators consume considerable CPU cycles when handling I/O-intensive workloads, leaving little headroom for growing user application requirements. Broadcom’s i. SCSI HBA architecture minimizes the CPU overhead so valuable CPU cycles are allocated to process user applications. Broadcom i. SCSI HBA only uses 1/3 of the CPU cycles required by the software initiator for a 4 K I/O size read operation” 11/3/2011 t. SAS/sreddy 14

t. SAS Approach § The t. SAS Request is initially sent by the t. SAS Initiator to the t. SAS Target over TCP. § The t. SAS Target strips off the TCP header and sends the SAS frames using the SAS Initiator block on the t. SAS Target to the SAS expander/drive. § The SAS expander/drive sends SAS frames to the t. SAS Target. § Finally, the t. SAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the t. SAS Initiator. 11/3/2011 t. SAS/sreddy 15

t. SAS Message Format Ethernet IP Header TCP Header t. SAS Data Ethernet Trailer TCP Segment IP Datagram Ethernet Frame § The above shows how a legacy SAS header and data are embedded in an Ethernet frame. § i. SCSI uses the same approach where the i. SCSI header and data are encapsulated in an Ethernet frame. 11/3/2011 t. SAS/sreddy 16

t. SAS Approach SAS Expander t. SAS Target t. SAS Initiator t. SAS Request Open Address Frame Open Accept SAS/SSP Request Frame t. SAS. Xfer Rdy Frame. Data Frame t. SAS Response Frame Xfer Rdy Frame. . Data Frame SSP Response Frame § The above diagram shows a typical t. SAS (SSP Write ) Request & Response sequence diagram. § i. SCSI uses a similar approach where the back-end between the i. SCSI target and SCSI drives uses the legacy 17 SCSI protocol. t. SAS/sreddy 11/3/2011

t. SAS Approach 11/3/2011 § Primitives − Primitives are special 8 b/10 b encoded characters that are used as frame delimiters, for out of band signaling, control sequencing in SAS − Most primitives can be conveniently ignored on the t. SAS side except a few such as Broadcast primitives used to initiate discovery − SAS primitive can be encapsulated in an Ethernet frame § Discovery − Discovery in t. SAS will be similar to SAS and will be accomplished by sending Serial management protocol (SMP) commands over TCP to the initiators and expanders downstream to learn the topology. t. SAS/sreddy 18

t. SAS Approach SAS Primitives 11/3/2011 t. SAS/sreddy 19

t. SAS Approach SMP Initiator on Expander t. SAS Client SMP Request SMP Target on Expander Open Address Frame Open Accept SMP Request SMP Response Close SMP Response SMP Request & Response Sequence Diagram 11/3/2011 t. SAS/sreddy 20

Test benches used for performance calculations § t. SAS and i. SCSI Mock Application − The client application and server application run on two different Windows machines connected using a Net. Gear Pro. Safe Gigabit Switch § Intel Core 2 Duo CPU § 2. 53 GHz, 3. 45 GB RAM § Intel 82567 LM Gigabit Network Adapter − Client makes Read/Write requests and the server processes and responds to requests made by the Client Windows Machine A (Running Client App) 11/3/2011 Net. Gear Switch t. SAS/sreddy Windows Machine B (Running Server App) 21

Test benches used for performance calculations § 11/3/2011 Legacy SAS − A 6 Gbps SAS Host Bus Adapter in a PCIe slot of a Super Micro server − A 6 Gbps SAS target (Seagate SAS drive) connected to the Host Bus Adapter − IOMeter in Windows and VDBench in RHEL used to generate Read/Write IOs and measure performance § IOMeter and VDBench are IO generator tools used to measure IO performance of a storage system − A Le. Croy SAS Protocol Analyzer placed between the HBA and the SAS Target t. SAS/sreddy 22

Test benches used for performance calculations SAS CONTROLLER (HOST BUS ADAPTER) Driver SAS Protocol Analyzer X SAS Protocol Analyzer - SAS DISK DRIVE X 11/3/2011 - EXPANDER t. SAS/sreddy 23

Test benches used for performance calculations § Legacy i. SCSI − Two windows machines/servers were used connected using a Net. Gear Pro. Safe Gigabit Switch − On one machine the Star. Wind i. SCSi Initiator was running − On another machine, the Kern. Safe i. SCSi target software was used to create a i. SCSI target − IOMeter was used to send Read/Write requests from the i. SCSI Initiator to the i. SCSI Target and measure performance − Wireshark used to capture Network Traffic i. SCSI Initiator on Windows Machine A 11/3/2011 Net. Gear Switch t. SAS/sreddy i. SCSI Target on Windows Machine B 24

t. SAS vs i. SCSI Mock Application at 10 Mbps i. SCSI vs t. SAS READ Completion Time at 10 Mbps 2500 2000 1500 Time (Milliseconds) 1000 t. SAS READ Completion time i. SCSI READ Completion Time 500 0 0 200 400 600 800 1000 Transfer Size (Kilobytes) 1200 t. SAS vs i. SCSI Write 10 Mbps 8000 7000 6000 5000 Time (milliseconds)4000 3000 2000 1000 0 t. SAS Write 10 MBps i. SCSI Write 10 MBps 0 11/3/2011 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 Transfer Size (Kilobytes) t. SAS/sreddy 25

t. SAS vs i. SCSI Mock Application at 100 Mbps t. SAS vs i. SCSI Read 100 Mbps 700 600 500 400 Time (Milliseconds)300 t. SAS Read 100 Mbps 200 i. SCSI Read 100 Mbps 100 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 Transfer Length (Kilobytes) t. SAS vs i. SCSi Write 100 Mbps 900 800 700 600 500 Time (Milliseconds)400 300 200 100 0 t. SAS Write 100 Mbps i. SCSI Write 100 Mbps 0 11/3/2011 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 Transfer Length (KB) t. SAS/sreddy 26

t. SAS vs i. SCSI Mock Application at 1 Gbps t. SAS vs i. SCSI Read 1 Gbps 50 45 40 35 30 Time 25 (Milliseconds)20 15 10 5 0 t. SAS Read 1 Gbps i. SCSi Read 1 Gbps 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 Transfer Length (KB) t. SAS vs i. SCSI Write 1 Gbps 80 70 60 50 Time 40 (Milliseconds) 30 t. SAS Write 1 Gbps i. SCSI Write 1 Gbps 20 10 0 0 11/3/2011 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 Transfer Size (KB) t. SAS/sreddy 27

t. SAS 512 K Read at 10 Mbps, 100 Mbps and 1 Gbps Time for READ Completion with transfer size of 512 K at 10 Mbps, 100 Mbps and 1 Gbps 1200 1000 800 Time (Milliseconds) 600 Time for READ Compeltion 400 200 0 0 11/3/2011 200 400 600 800 1000 Connection Rate (Mbps) t. SAS/sreddy 1200 28

Lessons Learnt/Difficulties encountered § Using several i. SCSI Target Softwares that were not licensed/purchased made it challenging since several features were either not supported or did not work on the unlicensed version. § Even using Kern. Safe i. SCSI target, I was not able to create a i. SCSI Ram. Disk Target or a i. SCSI Hard disk target. I used a USB Flash drive as the i. SCSI target − Subtracted the delay at the Flash Drive from the i. SCSI IOMeter performance results − Delay of Flash Drive calculated by running IOMeter to the SCSI device on the target machine 11/3/2011 29 t. SAS/sreddy

Lessons Learnt/Difficulties encountered § § 11/3/2011 Jumbo frames § The Net. Gear Pro. Safe switch is a Gigabit switch that supports Jumbo Frames. However, I could not get jumbo frames to work since the Network Adapter on the Windows machines did not support Jumbo frames § One needs cabling and adapters as well that support Jumbo frames to use a switch that supports Jumbo frames Understanding the SAS protocol and tools is very involved − Took substantial time to understand learn the workings of SAS as well as the tools t. SAS/sreddy 30

Future Work 11/3/2011 § More data points − The t. SAS mock application can be run using a faster switch with connection rates greater than 1 Gbps to get more data points (40 G/100 G Ethernet) § Piggybacking − Response frames are piggybacked with the last DATA frame or a DATA frame is piggy backed with a request frame. This would increase performance § Jumbo frames − Can be used to increase the amount of DATA that is passed from the initiator and target per Ethernet packet improving the performance results. § The Storage Associations can be motivated with similar work to fuel the conception of a t. SAS specification § t. SAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation t. SAS/sreddy 31

Conclusion § t. SAS is a viable solution § No conclusions can be made regarding latency of t. SAS compared to i. SCSI. However at 1 Gbps, t. SAS performs faster than i. SCSI on WRITEs § t. SAS more reliable at smaller transfer sizes at 10 Mbps, 100 Mbps and 1 Gbps due to higher TCP retransmission rate at larger sizes § Overhead in t. SAS smaller than in i. SCSI § Request, Response and Transfer Ready frames in t. SAS smaller than Request, Response and Ready to Transfer frames in i. SCSI § t. SAS will overcome the distance limitation of legacy SAS § Back-end of t. SAS will be a SAS/SATA drive which are faster 32 11/3/2011 t. SAS/sreddy

References [1] T 10/1760 -D Information Technology – Serial Attached SCSI – 2 (SAS-2), T 10, 18 April 2009, Available from http: //www. t 10. org/drafts. htm#SCSI 3_SAS [2] Harry Mason, Serial attached SCSI Establishes its Position in the Enterprise, LSI Corporation, available from http: //www. scsita. org/aboutscsi/sas/6 Gbps. SAS. pdf [3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner, RFC 3720 Internet Small Computer Systems Interface (i. SCSI) Specification IETF, April 2004, available from http: //www. ietf. org/rfc 3720. txt [4] Cai, Y. ; Fang, L. ; Ratemo, R. ; Liu, J. ; Gross, K. ; Kozma, M. ; A test case for 3 Gbps serial attached SCSI (SAS) Test Conference, 2005. Proceedings. ITC 2005. IEEE International, February 2006, available from http: //ieeexplore. ieee. org/xpls/abs_all. jsp? arnumber=1584027 11/3/2011 t. SAS/sreddy 33

References [6] Kalmath Meth, Julian Satran, Design of the i. SCSI Protocol, Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. 20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, available from http: //ieeexplore. ieee. org/xpls/abs_all. jsp? arnumber=1194848&tag=1 [7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke, A Performance Analysis of the i. SCSI Protocol 20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, available from http: //www. storageconference. org/2003/papers/20 -Aikens-Performance. pdf [8] M. Rajagopal, E. Rodriguez, R. Weber, RFC 3821 Fibre Channel over TCP/IP (FCIP) standard, IETF, July 2004, available from http: //tools. ietf. org/html/rfc 3821 [9] BLi Bigang, Shu Jiwu, Zheng Weimin, SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS * Department of Computer Science and Technology, Tsinghua University, Beijing China, 2005 11/3/2011 t. SAS/sreddy 34

References [10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. Mc. Daniel, M. Vilayannur, Design, Implementation and Evaluation of Security in i. SCSI-based Network Storage Systems, Storage. SS '06 Proceedings of the second ACM workshop on Storage security and survivability, October 2006, available from http: //portal. acm. org/citation. cfm? id=1179564 [11] 1 Gbps and 10 Gbps Ethernet Server Scalability, Net. App, available fromhttp: //partners. netapp. com/go/techontap/matl/downloads/redhat- neterion_10 g. pdf [12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM INTERFACE (SAS) v. OVER ETHERNET, United States Patent Application 20080228897, 09/18/2008 available from http: //www. faqs. org/patents/app/20080228897 [13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T 10, 2011/06/13, available from http: //www. t 10. org/members/w_spc 4. htm [14] Broadcom i. SCSI HBA FAQ available from http: //www. broadcom. com/collateral/faq/i. SCSI-FAQ 100 -R. pdf 11/3/2011 t. SAS/sreddy 35