The NVMe Management Interface NVMeMI Overview and New

The NVMe Management Interface (NVMe-MI): Overview and New Developments Peter Onufryk Microsemi NVMe Developer Days 2018 San Diego, CA 1

Agenda § § NVMe-MI 1. 0 a Overview New Features in NVMe-MI 1. 1 NVMe-MI Status Summary NVMe Developer Days 2018 San Diego, CA 2

NVM Express, Inc. 120 Companies Defining NVMe Together NVMe Developer Days 2018 San Diego, CA 3

What is the NVMe Management Interface 1. 0 a? A programmable interface that allows out-of-band management of an NVMe Storage Device Field Replaceable Unit NVMe Developer Days 2018 San Diego, CA 4

NVMe Storage Device Management Server Caching Root Complex NVMe x 16 NVMe SAS NVMe Controller B x 16 PCIe Switch x 4 NVMe Root Complex x 16 PCIe/PCI e RAID x 4 NVMe Controller A x 16 PCIe Switch NVMe External Storage Server Storage PCIe Switch SAS NVMe SAS HDD § Example Pre-boot Management • § Inventory, Power Budgeting, Configuration, Firmware Update Example Out-of-Band Management During System Operation • Health Monitoring, Power/Thermal Management, Firmware Update, Configuration Flash Memory Summit 2014 Santa Clara, CA 5

Why Standardize NVMe Storage Device Management? § Reduces Cost and Broadens Adoption • • § Consistent Feature Set • • § Allows OEMs to source storage devices from multiple suppliers Eliminates need for NVMe storage device suppliers to develop custom OEM specific management features All storage devices that implement management implement a common baseline feature set Optional features are implemented in a consistent manner Industry Ecosystem • • Compliance tests / program Development tools Flash Memory Summit 2014 Santa Clara, CA 6

NVMe Architecture NVM Subsystem - one or more controllers, one or more namespaces, one or more PCI Express ports, a non-volatile memory storage medium, and an interface between the controller(s) and non-volatile memory storage medium NVM Subsystem with One Controller and One Port NVMe Developer Days 2018 San Diego, CA NVM Subsystem with Two Controllers and Two Ports 7

NVMe Storage Device in NVMe-MI 1. 0 a NVMe Storage Device – One NVM Subsystem with one or more ports, optional FRU Information Device, and an optional SMBus/I 2 C interface NVMe Developer Days 2018 San Diego, CA 8

Vital Product Data (VPD) • • Utilizes IPMI Platform Management FRU Information Storage Definition with NVMe-MI extensions The VPD may be accessed using two methods – NVMe-MI commands over MCTP – SMBus/I 2 C interface using I 2 C operations as defined by IMPI Platform Management FRU Information Storage Definition VPD Elements Common Header Product Info Area (optional) NVMe Multi. Record Area NVMe PCIe Port Multi. Record Area Internal Use Area (optional) Chassis Info Area (optional) NVMe Developer Days 2018 San Diego, CA Board Info Area (optional) 9

Out-of-Band Management and NVMe-MI • • Out-of-Band Management – Management that operates with hardware resources and components that are independent of the operation system control NVMe Out-of-Band Management Interfaces – SMBus/I 2 C – PCIe Vendor Defined Messages (VDM) – IPMI FRU Data (VPD) accessed over SMBus/I 2 C NVMe Developer Days 2018 San Diego, CA 10

NVMe-MI Protocol Layering NVMe Developer Days 2018 San Diego, CA 11

NVMe-MI 1. 0 a Command Set Overview Command Type NVMe Management Interface Specific Commands PCIe Command NVMe Developer Days 2018 San Diego, CA Command Read NVMe-MI Data Structure NVM Subsystem Health Status Poll Controller Health Status Poll Configuration Get Configuration Set VPD Read VPD Write Reset Vendor Specific PCIe Configuration Read PCIe Configuration write PCIe I/O Read PCIe I/O Write PCIe Memory Read PCIe Memory Write Vendor Specific Command Type Command NVMe Admin Commands Firmware Activate/Commit Firmware Image Download Format NVM Get Features Get Log Page Identify Namespace Management Namespace Attachment Security Send Security Receive Set Features Vendor Specific 12

New Features in NVMe-MI 1. 1 § In-Band NVMe-MI § Enclosure Management § NVMe Storage Device Extensions NVMe Developer Days 2018 San Diego, CA 13

In-Band Management and NVMe-MI • In-band mechanism allows application to tunnel NVMe-MI commands through NVMe driver – Two new NVMe Admin commands – – • NVMe-MI Send NVMe-MI Receive Benefits – Provides management capabilities not available in-band via NVMe commands – Efficient NVM Subsystem health status reporting – Ability to manage NVMe at a FRU level – Vital Product Data (VPD) access – Enclosure management NVMe Developer Days 2018 San Diego, CA 14

NVMe JBOFs Facebook Lightning PCIe NVMe JBOF NVMe Developer Days 2018 San Diego, CA 15

NVMe Enclosure Management • Native PCIe Enclosure Management (NPEM) • • Submitted to the PCI-SIG Protocol Workgroup (PWG) on behalf of the NVMe™ Management Interface (NVMe-MI™) Workgroup Approved by PCI-SIG on August 10 th, 2017 Transport specific basic enclosure management SCSI Enclosure Services (SES) Based Enclosure Management • • Technical proposal developed in the NVMe-MI workgroup While the NVMe and SCSI architectures differ, the elements of an enclosure and capabilities to manage them are the same • • Example enclosure elements: power supplies, fans, display or indicators, locks, temperature sensors, current sensors, voltage sensors, and ports Comprehensive enclosure management for NVMe that leverages (SES), a standard developed by T 10 for management of enclosures using the SCSI architecture NVMe Developer Days 2018 San Diego, CA 16

NVMe Storage Device in NVMe-MI 1. 0 a NVMe Storage Device – One NVM Subsystem with one or more ports, optional FRU Information Device, and an optional SMBus/I 2 C interface NVMe Developer Days 2018 San Diego, CA 17

NVMe Storage Devices in NVMe-MI 1. 1 M. 2 Carrier Board from Amfeltec NVMe Developer Days 2018 San Diego, CA ANA Carrier Board from Facebook 18

NVMe-MI 1. 1 Storage Device Extensions NVMe-MI 1. 0 a NVMe Storage Device Definition An NVMe Storage Device consists of one NVM Subsystem with one or more ports, optional FRU Information Device, and an optional SMBus/I 2 C interface NVMe-MI 1. 1 An NVMe Storage Device contains at least one NVM subsystem or Expansion Connector and at least one Upstream Connector. It may contain additional elements such as: FRU Information Devices, PCIe switches, and SMBus/I 2 C Muxes. Supports Multiple NVM subsystems No Yes Supports Carriers No Yes Elements NVM Subsystem and FRU Information Device NVM Subsystem, Expansion Connector, Upstream Connector, FRU Information Device, PCIe Switch, and SMBus/I 2 C Mux VPD Describes basic characteristics of NVMe Storage Device Describes characteristics of NVMe Storage Device, elements that make up the NVMe Storage Device, and the element topology NVMe Developer Days 2018 San Diego, CA 19

NVM Express Roadmap NVMe Developer Days 2018 San Diego, CA 20

NVMe-MI Ecosystem § Commercial test equipment and conformance tests exist for NVMe-MI § NVMe-MI 1. 0 a compliance testing program exists • Compliance testing started in the May 2017 NVMe Plugfest conducted by the University of New Hampshire Interoperability Laboratory (UNH-IOL) • Multiple devices from multiple vendors have passed compliance testing and are on the NVMe-MI Integrators List § Servers are shipping that support NVMe-MI NVMe Developer Days 2018 San Diego, CA 21

Summary § NVMe-MI 1. 0 a has been released • Focused on managing NVMe Storage Devices (e. g. , SSDs) • SSDs and systems are shipping that support NVMe-MI 1. 0 a § NVMe-MI 1. 1 is nearing completion • Technical work is completed and a ratified specification is expected in Q 1’ 19 • Key new features in NVMe-MI 1. 1 – In-band NVMe-MI – Enclosure Management – NVMe Storage Device Extensions NVMe Developer Days 2018 San Diego, CA 22