STGTi SER Target Overview Or Gerlitz Voltaire ogerlitzvoltaire

STGT/i. SER Target Overview Or Gerlitz Voltaire ogerlitz@voltaire. com

agenda Ø some background Ø general structure / architecture Ø target drivers Øi. SCSI target & i. SCSI RDMA (i. SER) transport Ø SCSI devices Ø using / configuring STGT Ø some performance numbers Ø STGT community 2

some background Ø framework for SCSI protocol independent target Ø open source Ø STGT implemented / maintained by Fujita Tomonori & Mike Christie Ø i. SCSI RDMA (i. SER) transport implemented by Ohio Supercomputer Center staff under the direction of Pete Wyckoff 3

general structure / architecture The architecture is made of three elements: Ø 1 st : the SCSI state machine, I/O execution and management interface which are: Ø done in a SCSI protocol independent manner Ø pushed to user space, executed by tgtd user space daemon Ø 2 nd : target drivers - network interface Ø 3 rd : SCSI devices - storage interface Ø the Linux kernel was enhanced (2. 6. 20) to allow STGT use kernel target drivers 4

target drivers Ø user space – i. SCSI, FCo. E Ø kernel space – FC, IBMVIO Ø target driver API to tgt core Ø target create / destroy / update / show Ø lu create / lun get Ø notify on end of command / management 5

i. SCSI target driver Ø has i. SCSI transport independent state-machine Ø utilizes i. SCSI transport “library” Ø current transports: TCP and RDMA (i. SER) Ø i. SCSI transport API to the i. SCSI target driver Ø uses “end-point” notation to describe a connection Ø end-point read / write / rdma-read / rdma-write Ø end-point (implicit accept) / close / release / show Ø data buf alloc / free 6

i. SCSI RDMA (i. SER) transport Ø implemented over libibverbs and librdmacm Ø direct access to the RDMA HW device Ø RDMA transport neutral*, adopted to the Linux initiator Ø data transfer Ø uses RC QP (Queue Pair) Ø RDMA read (SCSI write), RDMA write (SCSI read) Ø Send/Receive for the SCSI commands request / response, management (eg login, nops), i. SCSI immediate data / unsolicited data-outs PDUs, etc 7

i. SER transport – main design issues Ø memory registration Ø register memory in advance to avoid performance drop Ø the target doesn’t advertise rdma keys hence a pool can be used for multiple clients Ø event management Ø adopt to a design that relies on socket readability / writeability (request notification, process completion) 8

SCSI devices Ø SBC - block command processing Ø OSD - object storage device command processing Ø also Ø SCC - controller command processing Ø MMC - multimedia command processing Ø SMC - Medium Changer command processing Ø SCSI device API to stgt core Ø lu init / config / exit Ø device ops (eg TUR, INQ, SPACE, READ*, WRITE*, SEEK) Ø below SCSI devices a “backing store” layer further exists Ø some possible types: aio, mmap, sync file 9

STGT package (RPM) Ø tgtd – service Ø tgtd - daemon Ø tgtadm – admin tool Ø tgt-setup-lun – helper/simper admin tool (uses tgtadm) Ø man pages to the admin tools 10

configuring STGT # tgt-setup-lun -d /dev/sdy -n seed 1 -sdy 192. 168. 10. 81 one liner setup which translates to the following sequence with tgtadm Ø create a target # tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn. 2001…-seed 1 -sdy Ø add a logical unit to the target # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sdy Ø bind initiators to this target instance # tgtadm --lld iscsi --op bind --mode target --tid 1 -I 192. 168. 10. 81 11

configuring STGT - cont’ # tgtadm --mode target --op show Target 1: iqn. 2001 -04. com. onion-seed 1 -sdy System information: Driver: iscsi Status: running LUN information: LUN: 0 Type: controller (this LUN is skipped here) LUN: 1 Type: disk SCSI ID: deadbeaf 1: 1 SCSI SN: beaf 11 Size: 82 G Online: Yes Poweron/Reset: Yes Backing store: /dev/sdy ACL information: 192. 168. 10. 81 12

some performance numbers Ø over ramdisk (single initiator, source Voltaire) Ø BW: READ – 1200 MB/sec WRITE – 870 MB/sec Ø IOPS (I/O per second): READ – 62 K WRITE – 66 K Ø over ramdisk (source Pete Wyckoff et al) Ø Ø latency for some OSD ops: ping 33 us, getattr – 65 us single initiator BW : READ – 550 MB/sec WRITE – 500 MB/sec multiple initiators BW: both READ/WRITE reach 900 MB/sec few variations between READ / WRITE Ø what message size gives the max BW (read 200 KB, write 500 KB) Ø how many initiators needed to get the aggregated BW under specific message size: READ – 2 WRITE – 8 13

STGT / community Øweb site http: //stgt. berlios. de has pointers to: Ø git tree where tgt is maintained Ø developers mailing list Ø wiki @ open-fabrics https: //wiki. openfabrics. org/tikiindex. php? page=ISER-target Ø explain / example usage Ø download packages for RH 5 U 1 and SLES 10 SP 1 14

references Ø tgt: framework for storage target drivers Fujita Tomonori and Mike Christie. Ottawa Linux Symposium 2006 Ø i. SER Storage Target for Object-based Storage Devices Dennis Dalessandro, Ananth Devulapalli, Pete Wyckoff Proceedings of MSST'07, SNAPI Workshop, San Diego, CA, Sep 2007 Ø source tree @ http: //www. kernel. org/pub/scm/linux/kernel/git/tomo/tgt. git 15