Architected for Performance NVMe TM Management Interface NVMeMITM

Architected for Performance NVMe. TM Management Interface (NVMe-MITM) Workgroup Update Sponsored by NVM Express™ organization, the owner of NVMe™, NVMe-o. F™ and NVMe-MI™ standards

Speaker Austin Bolen 2

Agenda § NVMe-MITM Workgroup Update § Management Fundamentals § NVMe-MI 1. 0/1. 0 a and 1. 1 Overview § What’s Coming in 1. 2 § Summary 3

NVM Express. TM, Inc. 120+ Companies defining NVMe together Marketing Workgroup Board of Directors NVMexpress. org, webcasts, tradeshows, social media, and press Co-Chairs: Cameron Brett and Jonmichael Hands 13 elected companies, stewards of the technology & driving processes Chair: Amber Huffman Technical Workgroup NVMe Base and NVMe Over Fabrics Chair: Peter Onufryk Management Interface Interop (ICC) Workgroup Interop & Conformance Testing Systems Management of NVMe Chair: Peter Onufryk Vice Chair: Austin Bolen in collaboration with UNH-IOL Chair: Ryan Holmqvist 4

NVM Express. TM, Inc. Roadmap 2014 Q 2 Q 3 2015 Q 4 Q 1 Q 2 Q 3 2016 Q 4 Q 1 Q 2 Q 3 2017 Q 4 Q 1 Q 2 Q 3 2018 Q 4 NVMe 1. 3 May’ 17 NVMe 1. 2. 1 May’ 16 NVMe. TM 1. 2 Nov’ 14 • Sanitize • Streams • Virtualization • Namespace Management • Controller Memory Buffer • Host Memory Buffer • Live Firmware Update NVMe Over Fabrics (NVMe-o. F) NVM Express (NVMe) Q 1 NVMe-o. FTM 1. 0 May’ 16 • • Q 1 Q 2 2019 Q 3 Q 4 Q 1 Q 2 Q 3 Q 4 NVMe 1. 4 Jun’ 19 • IO Determinism • Persistent Memory Region • Persistent Event Log • Multipathing NVMe-o. F (next) Transport and protocol RDMA binding • Enhanced Discovery • In-band Authentication • TCP Transport NVMe Mgmt. Interface (NVMe-MI) Binding • • NVMe-MITM 1. 0 Nov’ 15 Out-of-band management Device discovery Health & temp monitoring Firmware Update Released NVMe Specification NVMe-MITM 1. 0 a Apr’ 17 NVMe-MI 1. 1 Apr’ 19 • In-band Mechanism • Enclosure Management • Support for new Device Types - Multi-NVM Subsystem - Carrier Cards Planned Release* * Subject to change 5

Management Fundamentals • • • Management Operational Times Deployment (No OS) Pre-OS (e. g. , UEFI/BIOS) Runtime Auxiliary Power Decommissioning Pillars of Systems Management • • Inventorying Configuration Monitoring Change Management 6

The Importance of Manageability in Servers Availability A measure of system uptime. Unreliable components, inability to manage, and inability to service reduce availability. Reliability A measure of the reliability of systems and components. Better reliability increases availability and reduces replacement cost. Serviceability RASM A measure of how easy it is to recover a system to full operation following a failure. Manageability A measure of how easy it is to inventory systems/components, configure, perform updates, and monitor and report failures. “Customers choose suppliers who provide the features that are important to them. Customers care about TCO (Total Cost of Ownership). Consequently, in the server space, MHz is not the only thing that's important: TCO is greatly affected by the RASM features of the servers. When server OEMs and users talk, their focus is RASM: Reliability, Availability, Serviceability, and Manageability. To a customer, RASM means dollars. Adding or improving on RASM reduces TCO. The cost of downtime is extremely high. According to IMEX Research, the average cost of an unplanned outage runs into the hundreds of thousand of dollars. ” Better RASM = Reduced TCO 7

Out-of-Band Management and NVMe-MITM in 1. 0 § Out-of-Band Management – Management that operates with hardware resources and components that are independent of the host operating system control § NVMe Out-of-Band Management Interfaces § SMBus/I 2 C § PCIe Vendor Defined Messages (VDM) 8

NVMe-MITM Out-of-Band Protocol Layering 9

In-Band Management and NVMe-MITM in 1. 1 § In-band mechanism allows application to tunnel NVMe-MITM commands through NVMe. TM driver § Two new NVMe Admin commands – – NVMe-MI Send NVMe-MI Receive § Benefits § Provides management capabilities not available in-band via NVMe. TM commands – Efficient NVM Subsystem health status reporting – Ability to manage NVMe at a FRU level – Vital Product Data (VPD) access – Enclosure management 10

NVMe-MITM over NVMe. TM over Fabrics (NVMe-o. FTM) Plumbing in place for NVMe-MITM over NVMe-o. FTM 11

NVMe. TM Storage Device in 1. 0 § NVMe. TM Storage Device – One NVM Subsystem with one or more ports, vital product data (VPD), and an optional SMBus/I 2 C interface 12

Support for Complex NVMe. TM Storage Devices in 1. 1 ANA Carrier Board from Facebook NVMe Storage Device with two NVM Subsystems and an SMBus/I 2 C Mux NVMe. TM Storage Device with Expansion Connectors (i. e. , a Carrier) M. 2 Carrier Board from Amfeltec § NVMe Storage Device – A logical or physical component, device, or assembly that contains at least one NVM Subsystem or Expansion Connector, at least one Upstream Connector, and at least one FRU Information Device. An NVMe Storage Device that implements the out-of-band mechanism contains at least one Management Endpoint and a Controller Management Interface per Controller. An NVMe Storage Device contains zero or more PCIe switches and SMBus/I 2 C Muxes. 13

NVMe. TM Enclosure Management in 1. 1 § SES Based Enclosure Management § Technical proposal developed in NVMe-MITM workgroup § While the NVMe. TM and SCSI architectures differ, the elements of an enclosure and the capabilities required to manage these elements are the same – Example enclosure elements: power supplies, fans, display or indicators, locks, temperature sensors, current sensors, voltage sensors, and ports § Comprehensive enclosure management that leverages SCSI Enclosure Services (SES), a standard developed by T 10 for management of enclosures using the SCSI architecture 14

NVMe. TM Basic Management Command Deprecated in 1. 1 § The NVMe. TM Basic Management Command was defined as a whitepaper to allow rudimentary out-of-band management of NVMe as a stop-gap until the NVMe-MITM 1. 0 Specification was released § NVMe Basic Management Command is defined in an informative appendix in the NVMe-MI Specification, but is NOT part of NVMe-MI § The NVMe-MI Workgroup and Board have agreed that there will be no further enhancements to the NVMe Basic Management Command § It is strongly recommended that any consumers of the NVMe Basic Management Command transition to using the standard NVMe-MI protocol § The NVMe-MI Workgroup can help anyone currently using the NVMe Basic Management Command that feels that they cannot transition to NVMe-MI for technical reasons 15

Features in Development for NVMe-MITM 1. 2 § Routine Maintenance Updates § Sync to NVM Express. TM Base Specification 1. 4 § Errata fixes § New Features § Additional host metadata § Configuration of NVMe-o. FTM Exported NVMe. TM Resources § VPD size increase § Command Initiated Auto Pause § Industry Alignment § Alignment with DMTF and SNIA storage management 16

Additional Controller Metadata Element Types in 1. 2 Value Definition 00 h Reserved 01 h 02 h § Failure analysis of an SSD relies upon knowing the environment in which it failed § This feature adds additional information about the host operating environment such as system product name, processor information, etc. § Also adds ability for a host to log the reason why it reported a drive as failed. § Persistent Event Log feature defined by the NVMe. TM Base Specification stores the information for later retrieval (e. g. , when a failed drive is returned for failure analysis). 03 h 04 h 05 h 06 h 07 h 08 h 09 h 0 Ah 0 Bh 0 Ch 0 Dh 0 Eh 0 Fh Operating System Controller Name: The name of the Controller in the operating system as a UTF-8 string. Operating System Driver Name: The name of the driver in the operating system as a UTF-8 string. Operating System Driver Version: The version of the driver in the operating system as a UTF-8 string. Pre-boot Controller Name: The name of the Controller in the preboot environment as a UTF-8 string. Pre-boot Driver Name: The name of the driver in the pre-boot environment as a UTF-8 string. Pre-boot Driver Version: The version of the driver in the pre-boot environment as a UTF-8 string. System Processor Model: The model of the CPU as a UTF-8 string. Chipset Driver Version: The chipset driver version as a UTF-8 string. Operating System Name and Build: The operating system name and build as a UTF-8 string. System Product Name: The system product name as a UTF-8 string. Firmware Version: The host firmware (e. g. , UEFI) version as a UTF-8 string. Operating System Driver Filename: The operating system driver filename as a UTF-8 string. Display Driver Name: The display driver name as a UTF-8 string. Display Driver Version: The display driver version as a UTF-8 string. Host-Determined Failure Record: A failure record (e. g. , reason for NVM subsystem return) as a UTF-8 string. 17

Command Initiated Auto Pause in 1. 2 § NVMe-MITM has control primitives that allow pausing and resuming of message responses § In a segmented SMBus/I 2 C environment it may be convenient to automatically pause a response when a command is issued § This feature defines a mechanism that allows a pause to be automatically initiated when a command is issued 18

Vital Product Data (VPD) Size Increase in 1. 2 § VPD contains information about the physical storage device or Field Replaceable Unit (FRU) such as serial number, connectivity details, power requirements, etc. § VPD builds on top of IPMI Platform Management FRU Information Storage Definition which limits size to 256 bytes § Complex devices require more than 256 bytes of VPD § This feature will allow VPD size to increase in size up to 4 Ki. B 19

Configuration of NVMe-o. FTM Exported NVMe. TM Resources in 1. 2 Base NVM Subsystem (e. g. , a JBOF) Physical/Logical Namespaces Base Namespaces § NVMe-MITM issues commands to an Admin Controller in an NVMe-o. FTM system called a Base NVM Subsystem (e. g. , a JBOF) Physical Storage § New NVMe-MI commands: § Discover backend storage Namespaces (Base Namespaces) Exported Logical NVM Subsystem (SUBNQN) Exported Logical Namespaces Allowed Host NQNs Exported Logical Port ID Admin Ctrl Supporting MI Base Port (Ethernet) NVMe-MI Utility Known Host List § Discover physical fabrics ports (Base Ports) § Provision Base Namespaces into logical NVM Subsystems (Exported Logical NVM Subsystem) § Expose Exported Logical NVM Subsystems over specified Base Ports § Configure which hosts have access to Exported Logical NVM Subsystems Base Port (Fibre Channel) Base Port (Infini. Band) 20

Redfish Enhancements for NVMe-MITM DMTF, NVMe. TM and SNIA Form Three-Way Alliance The Distributed Management Task Force (DMTF), NVM Express TM, Inc. and Storage Networking Industry Association (SNIA) have formed a new three-way alliance to coordinate standards for managing solid state drive (SSD) storage devices. DMTF and SNIA have worked together as part of the DMTF’s Alliance Partner program for many years. Most recently, SNIA’s Swordfish™ standard is building upon DMTF’s Redfish® API to address enterprise storage services. NVM Express is an open collection of standards and information to fully expose the benefits of non-volatile memory (NVM) in all types of computing environments from mobile to data center. The three-way alliance will work on scale out management for SSDs, helping enable a holistic management experience. In addition to SNIA’s Swordfish and DMTF’s Redfish, the alliance’s collaborative work will include the following standards: § NVM Express™ (NVMe™) is the register interface and command set for PCI Express attached storage with industry standard software available for numerous operating systems. The NVM Express™ Management Interface (NVMe. MI™) is the command set and architecture for management of NVM Express storage (e. g. , discovering, monitoring, and updating NVMe devices using a BMC). § DMTF’s Management Component Transport Protocol (MCTP) is a protocol and Platform Level Data Model (PLDM) is a low-level data model defined by the DMTF Platform Management Components Intercommunications (PMCI) Working Group. MCTP is designed to support communications between different intelligent hardware components that make up a platform management subsystem that provides monitoring and control functions inside a managed system. § DMTF’s PLDM for Redfish Device Enablement (RDE) defines messages and data structures used for enabling PLDM devices to participate in Redfish-based management without needing to support either Java. Script Object Notation (JSON, used for operation data payloads) or the [Secure] Hypertext Transfer Protocol (HTTP/HTTPS, used to transport and configure operations). 21

Summary § NVMe-MITM drives and systems are shipping § NVMe-MI 1. 1 released in April § NVMe-MI 1. 2 development is underway 22

Additional Material on NVMe-MITM • • • Bright. TALK Webinar o https: //www. brighttalk. com/webcast/12367/282765/the-nvme-management-interface-nvme-mi-learn-whatsnew Conference Con. Cepts’ NVMe Developer Days – Dec. 2018 o Slides: https: //nvmexpress. org/wp-content/uploads/NVMe-Developer-Days-%E 2%80%93 -December-2018%E 2%80%93 -The-NVMe-Managemant-Interface-NVMe-MI-%E 2%80%93 -Overview-and-New. Developments. pptx Flash Memory Summit 2017 o Slides: https: //www. flashmemorysummit. com/English/Collaterals/Proceedings/20170808_FA 12_Part. A. pdf o Video: o https: //www. youtube. com/watch? v=da. KL 7 t. Iv. NII o https: //www. youtube. com/watch? v=Daqj-Xql. Co 8 Flash Memory Summit 2015 o Slides: https: //www. flashmemorysummit. com/English/Collaterals/Proceedings/20150811_FA 11_Carroll. pdf Flash Memory Summit 2014 o Slides: https: //www. flashmemorysummit. com/English/Collaterals/Proceedings/20140804_Seminar. F_Onufryk_ Bolen. pdf NVMe-MI Specification o https: //nvmexpress. org/resources/specifications/ 23

References MCTP Overview: http: //dmtf. org/sites/default/files/standards/documents/DSP 2016. pdf MCTP Base Spec: https: //www. dmtf. org/sites/default/files/standards/documents/DSP 0236_1. 3. 0. pdf MCTP SMBus/I 2 C Binding: https: //www. dmtf. org/sites/default/files/standards/documents/DSP 0237_1. 1. 0. pdf MCTP PCIe VDM Binding: https: //www. dmtf. org/sites/default/files/standards/documents/DSP 0238_1. 0. 2. pdf IPMI Platform Management FRU Information Storage Definition: https: //www. intel. la/content/www/xl/es/servers/ipmi-platform-mgt-fru-infostorage-def-v 1 -0 -rev-1 -3 spec-update. html NVM Express Base Specification 1. 4: https: //nvmexpress. org/wp-content/uploads/NVM-Express-1_42019. 06. 10 -Ratified. pdf 24

Architected for Performance