NVMe flash storage is coming to an enterprise near you
Hyperscale data centers are the early adopters of non-volatile memory express flash, as vendors lay out plans to further flash devices and NVMe fabrics in coming years.
NVMe flash storage is heading for broader enterprise use, but don't expect a wholesale transition to begin at least until 2019 -- and maybe later. That was the takeaway from a series of presentations at the inaugural NVMe Developer Days event in San Diego this month.
Geared mostly toward infrastructure and DevOps teams, the two-day event drilled into the nitty-gritty of NVMe storage -- sometimes so deeply, it left participants too dazed to ask questions during Q&A sessions.
Despite the developer-heavy focus, talk of storage products was woven into many of the engineering discussions. Canadian startup Eideticom showcased its U.2-based NVMe offload card for accelerating storage tasks that require dense computation. NGD Systems touted its computational storage Newport application-specific integrated circuit NVMe platform, which injects added computing power into storage to accelerate applications.
As interesting as these products are, they don't appear to be intended for enterprises, at least not initially. NVMe flash adoption is dominated by cloud service providers, high-frequency trading and hyperscale applications, as proposed industry standards for the high-speed storage transport inch toward ratification.
Despite increased demand for high-performance storage, most enterprises don't have a need to switch to NVMe flash storage yet, said Cisco R&D Engineer for Advanced Storage J Metz.
"NVMe as an interface, as well as a genre of storage, has changed the nature of two particular things," Metz said. "One is CPU utilization, and the other is I/O. Management of both is becoming something of an issue, to the point where the proper switching of PCIe becomes important. The proper use of interrupts versus polling on the CPU becomes important. The proper location of where the I/O is actually processed becomes important. Those all have to do with the direct tie-in to which hardware you're using and the communication between the hardware."
There are enterprise arrays on the market already, as the major storage vendors lay the groundwork for mass adoption. Dell EMC, Hewlett Packard Enterprise Nimble Storage, IBM, NetApp and Pure Storage are shipping all-flash arrays with NVMe media, usually along with SAS and SATA drives.
A handful of startups also sell arrays that exclusively use NVMe U.2 SSDs. The list includes E8 Storage, Apeiron Data Systems, Excelero, Exten -- formerly Mangstor -- Pavilion Data and Vexata. With the exception of Excelero, all of the startups package branded operating software on qualified servers equipped with NVMe SSDs.
Although NVMe is new to many data center professionals, its roots predate the widespread enterprise use of flash. The precursor to the NVMe standard first emerged in 2008. The nonvolatile memory host controller interface specification was created to give SSD makers a supported standard to develop PCI Express (PCIe) cards and drivers. That allows an application to avoid network hops and directly access the storage via a PCIe bus.
NVMe: A 'niche play' until fabrics mature
NVMe technology was designed from the ground up for flash storage. NVMe pushes flash storage closer to the motherboard, allowing storage and CPUs to connect with a PCI Express link. Leading enterprise array makers market NVMe all-flash systems that replace SAS and SATA SSDs with PCIe-connected NVMe flash drives.
Analyst firm IDC said NVMe flash storage is a $500 million market today, but will generate roughly half of all storage revenue by 2021.
The emergence of NVMe SSDs helped fuel scale-up storage in commodity servers and changes how enterprises integrate flash, said Fazil Osman, a distinguished engineer at Broadcom, which owns storage switch giant Brocade Communications.
"There will always be customers that want the box, especially when it comes to storage," Osman said. "The market is not going away, but the big storage vendors are looking at how they can sell more scale-up storage, especially for analytics and AI. The fact that all the major SSD vendors use a common NVMe driver is a big deal, because there's no need to add a driver to qualify a new vendor."
At this point, however, NVMe flash remains a "niche play" for most brownfield deployments, said Frederic Van Haren, a senior analyst in the high-performance computing and AI practice at Evaluator Group, based in Boulder, Colo.
"If you're at one of the stock exchanges, then for sure you're looking at NVMe flash to try and skim off a few microseconds of proximity on the fiber that goes to the trading floor," Van Haren said.
But, on average, an all-flash array typically exploits only 30% to 40% of its theoretical performance capabilities.
"I can replace six racks of spindles with half a rack of all-flash SAS SSDs and get the same performance as disk," Van Haren said. "With NVMe, you can show me there is a theoretical improvement by switching, but as an end user, I wouldn't notice it."
Industry specifications advance
NVM Express Inc., an industry consortium of more than 120 vendors, released preview of NVMe 1.4, a set of industry specifications scheduled for publication in 2019.
The standard includes more than 20 ratifications that component and drive makers may use to get a jump on building NVMe storage devices that maximize NAND flash, said Jonmichael Hands, an Intel product manager and marketing co-chair of the NVM Express group.
With NVMe 1.4, NVM Express released each technical proposal and feature as it got ratified, a departure from the previous versions. The roadmap includes efforts to improve I/O determinism through physical isolation of NAND, multipurpose persistent memory regions, NVMe multipathing and namespace sharing, and rebuild assist to enhance data integrity. Hands said full NVMe 1.4 specs will be available in the second half of 2019.
"We had complaints that it was taking vendors about two years to catch up to [the previous] generation of the spec. We now make each proposal available as it gets ratified, so system developers and SSD manufacturers can start developing the features and functionality" without waiting for the full specification to be approved, Hands said.
The forthcoming specs also aim to advance NVMe over fabrics (NVMe-oF) architectures to enable sharing of block-based NVMe flash storage devices across multiple servers. Most of the industry work is centered on legacy Fibre Channel or remote direct memory access (RDMA), although switch vendors could write drivers for other storage fabrics.
NVMe-oF gives the advantage of shared storage, better use of rack space and power. Companies can gain the advantage of NVMe now simply by swapping out SAS SSDs, but the biggest advantage is bringing storage and processing closer together.
"We've got faster wire speeds, but we need a faster protocol. That's where NVMe over fabrics comes in. The good thing is it isn't tied to any one wire; it can run across any fabric," Rob Davis, a Mellanox vice president of storage technology, told attendees during a presentation.
The Evaluator Group's Van Haren said the shakeout among various NVMe fabric transports bears watching for anyone managing storage, especially Ethernet-enabled RDMA and FC.
"A lot of effort is going into using existing [network] infrastructure," he said. "That means vendors are looking at easy portability of NVMe over fabrics" for use as mainstream data centers technology.
