Cinder and NVMeoverFabrics NetworkConnected SSDs with Local Performance

Cinder and NVMe-over-Fabrics Network-Connected SSDs with Local Performance Tushar Gohad, Intel Moshe Levi, Mellanox Ivan Kolodyazhny, Mirantis

Storage Evolution Technology claims are based on comparisons of latency, density and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications. 2

Intel 3 D XPoint* Performance at QD=1 3

NVM Express (NVMe) Standardized interface for non-volatile memory, http: //nvmexpress. org Source: Intel. Other names and brands are property of their respective owners. Technology claims are based on comparisons of latency, density and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications. 4

NVMe: Best-in-Class IOPS, Lower/Consistent Latency IOPS - 4 K Random Workloads 500000 PCIe/NVMe IOPS 400000 SAS 12 Gb/s SATA 6 Gb/s HE 300000 200000 100000 0 100% Read 70% Read 3 x better IOPS vs SAS 12 Gbps For the same #CPU cycles, NVMe delivers over 2 X the IOPs of SAS! Lowest Latency of Standard Storage Interfaces Gen 1 NVMe has 2 to 3 x better Latency Consistency vs SAS Test and System Configurations: PCI Express* (PCIe*)/NVM Express * (NVMe) Measurements made on Intel® Core™ i 7 -3770 S system @ 3. 1 GHz and 4 GB Mem running Windows * Server 2012 Standard O/S, Intel PCIe/NVMe SSDs, data collected by IOmeter * tool. SAS Measurements from HGST Ultrastar* SSD 800 M/1000 M (SAS), SATA S 3700 Series. For more complete information about performance and benchmark results, visit http: //www. intel. com/performance. Source: Intel Internal Testing. 5

Remote Access to Storage – i. SCSI and NVMe-o. F • NVMe-over-Fabrics Network Disaggregated Cloud Deployment Model • NVMe commands over storage networking fabric i. SCSI Target NVMe-o. F* Target SCSI NVMe Block Device Abstraction (BDEV) SCSI Devices NVMe Devices • NVMe-o. F supports various fabric transports • • • RDMA (Ro. CE, i. WARP) Infini. Band™ Fibre Channel Intel® Omni-Path Architecture Future Fabrics 6

NVMe and NVMe-o. F Basics

NVMe Subsystem Implementations including NVMe-o. F

NVMe-o. F: Local NVMe Performance • The idea is to extend the efficiency of the local NVMe interface over a network fabric • Ethernet or IB • NVMe commands and data structures are transferred end to end • Relies on RDMA for performance • Bypassing TCP/IP • For more Information on NVMe over Fabrics (NVMe-o. F) • http: //www. nvmexpress. org/wpcontent/uploads/NVMe_Over_Fabrics. pdf 9

What Is RDMA? § Remote Direct Memory Access (RDMA) § Advance transport protocol (same layer as TCP and UDP) § Main features § Remote memory read/write semantics in addition to send/receive § Kernel bypass / direct user space access § Full hardware offload § Secure, channel based IO § Application advantage § Low latency § High bandwidth § Low CPU consumption § Ro. CE, i. WARP § Verbs: RDMA SW interface (equivalent to sockets)

RDMA and NVMe: A Perfect Match Network 11

Mellanox Product Portfolio Switches NICs Cables Ethernet & Infini. Band RDMA End-to-End 25, 40, 56, 100 Gb 12

NVMe-o. F – Kernel Initiator • Uses nvme-cli package implement the kernel initiator side • Connect to remote target • nvme connect –t rdma –n <conn_nqn> –a <target_ip> –s <target_port> • nvme list - to get all the nvme devices 13

NVMe-o. F – Kernel Target • Uses nvmetcli package implement the kernel target side • nvme save <file_name>– to create new subsystem • nvme restore – to load existing subsystems 14

NVMe-o. F in • • Available from Rocky release (we hope ☺ ) Available with Triple. O deployment Requires RDMA NICs Supports Kernel target Supports Kernel Initiator SPDK target is work in progress Work Credit: • Ivan Kolodyazhny (Mirantis) – First POC with SPDK • Maciej Szwed (Intel) - SPDK Target • Hamdy Khadr, Moshe Levi (Mellanox) – Kernel Initiator and Target 15

NVMe-o. F in 16

Horizon Client NVMe-o. F in Cinder Nova NVMe-o. F Target Drv Tenant VM /dev/vda New Kernel LVM Volume Drv New LVM • First implementation of NVMe-over-Fabrics in Open. Stack • Target Open. Stack Release: Rocky (Logical Volume Manager) KVM NVMe-o. F Initiator RDMA Capable Network nvmet NVMe-o. F Target NVMe-o. F Data Path Nova/Cinder Control Path

NVMe. OF – Backend [nvme-backend] lvm_type = default volume_group = vg_nvme volume_driver = cinder. volume. drivers. lvm. LVMVolume. Driver volume_backend_name = nvme-backend target_helper = nvmet target_protocol = nvmet_rdma target_ip_address = 1. 1 target_port = 4420 nvmet_port_id = 2 nvmet_ns_is = 10 target_prefix = nvme-subsystem-1 18

NVMe. OF with # cat /home/stack/tripleo-heat-templates/environments/cinder-nvmeof-config. yaml parameter_defaults: Cinder. NVMe. OFBackend. Name: 'tripleo_nvmeof' Cinder. NVMe. OFTarget. Port: 4420 Cinder. NVMe. OFTarget. Helper: 'nvmet' Cinder. NVMe. OFTarget. Protocol: 'nvmet_rdma' Cinder. NVMe. OFTarget. Prefix: 'nvme-subsystem' Cinder. NVMe. OFTarget. Port. Id: 1 Cinder. NVMe. OFTarget. Name. Space. Id: 10 Controller. Parameters: Extra. Kernel. Modules: nvmet: {} nvmet-rdma: {} Compute. Parameters: Extra. Kernel. Modules: nvme: {} nvme-rdma: {} 19

NVMe-o. F and SPDK Storage Performance Development Kit

Scalable and Efficient Software Ingredients Storage Performance Development Kit • • • User space, lockless, polled-mode components Up to millions of IOPS per core Designed to extract maximum performance from nonvolatile media Storage Reference Architecture • Optimized for latest generation CPUs and SSDs • Open source composable building blocks (BSD licensed) • Available via spdk. io

Benefits of using SPDK

SPDK Architecture Storage Protocols NVMe-o. F* RDMA Target vhost-scsi Target i. SCSI Target vhost-blk Target Cinder VPP TCP/IP Block Device Abstraction (bdev) Logical Volumes 3 rd Party NVMe Linux AIO Ceph RBD NVMe Devices Drivers Linux nbd SCSI NVMe Storage Services Integration NVMe-o. F* Initiator NVMe* PCIe Driver GPT PMDK blk Qo. S DPDK Encryption virtio scsi virtio blk Rocks. DB Blob. FS Blobstore Ceph QEMU Core Intel® Quick. Data Technology Driver Application Framework

NVMe-o. F Performance with SPDK vs. Kernel NVMe-o. F I/O Efficiency 4, 000 35 3 x 50 Gb. E Line Rate 3, 500, 000 30 30 3, 000 25 2, 500, 000 20 2, 000 NVMe* over Fabrics Target Features Realized Benefit Utilizes NVM Express* (NVMe) Polled Mode Driver Reduced overhead per NVMe I/O RDMA Queue Pair Polling No interrupt overhead Connections pinned to CPU cores No synchronization overhead 15 1, 500, 000 10 1, 000 500, 000 5 3 0 0 SPDK Kernel IOPS Cores SPDK reduces NVMe over Fabrics software overhead up to 10 x! System Configuration: Target system: Supermicro SYS-2028 U-TN 24 R 4 T+, 2 x Intel® Xeon® E 5 -2699 v 4 (HT off), Intel® Speed Step enabled, Intel® Turbo Boost Technology enabled, 8 x 8 GB DDR 4 2133 MT/s, 1 DIMM per channel, 12 x Intel® P 3700 NVMe SSD (800 GB) per socket, -1 H 0 FW; Network: Mellanox * Connect. X-4 LX 2 x 25 Gb RDMA, direct connection between initiators and target; Initiator OS: Cent. OS * Linux* 7. 2, Linux kernel 4. 10. 0, Target OS (SPDK): Fedora 25, Linux kernel 4. 9. 11, Target OS (Linux kernel): Fedora 25, Linux kernel 4. 9. 11 Performance as measured by: fio, 4 KB Random Read I/O, 2 RDMA QP per remote SSD, Numjobs=4 per SSD, Queue Depth: 32/job. SPDK commit ID: 4163626 c 5 c

SPDK LVOL Backend for Openstack Cinder • First implementation of NVMe-over. Fabrics in Openstack • • NVMe-o. F Target Driver SPDK LVOL based SDS Storage Backend (Volume Driver) • Provides High-performance Alternative to Kernel LVM and Kernel NVMe-o. F Target • • • Upstream Cinder PR# 564229 Target Openstack Release: Rocky Joint work by Intel, Mirantis, Mellanox

Demonstration Upcoming Rocky NVMe-o. F Feature