NVMe over FC is happening, and it's time to prepare
The extent to which the recently published FC-NVMe standard lowers performance latencies in Fibre Channel SANs will be interesting to watch in the coming months.
Are you ready for the next wave of IT infrastructure? The Fibre Channel Industry Association recently published the industry's standard for nonvolatile memory express over FC. Part of the larger NVMe-over-Fabrics specification, NVMe over FC, also known as FC-NVMe, represents a critical next step in defining the future of the data center.
NVMe technology has been around for a while now. Unlike more traditional storage interface protocols, such as SAS or SATA, NVMe extends the lower-latency performance benefits of memory-based storage media like flash storage to the application. Developed to operate over a PCI Express transport layer, NVMe technology has been predominately limited to internal NVMe-based server SSDs or within direct-attached storage.
With the NVMe-over-FC standard, there's now an agreed upon methodology to use NVMe-based communication over a Fibre Channel SAN. This dramatically increases the accessibility of NVMe, improving the performance of FC SANs.
FC-NVMe and existing FC SAN infrastructure
FC-NVMe is an important milestone, but there's still work to do. The FC-NVMe standard is a huge step forward, but there's a difference between publishing a standard and creating an enterprise product that works end to end. To this end, the Fibre Channel Industry Association (FCIA) hosts multiple plugfest events to encourage FC technology vendors -- such as host bus adapter, switch and storage manufacturers -- to test their systems in multivendor environments to ensure everything works. While these activities are important, vendors also conduct their own product-level testing.
Additionally, FCIA is already working on FC-NVMe-2, adding transport error recovery at the FC level to avoid connection termination. FC-NVMe-2 likely improves the strength of FC-NVMe systems, making them better able to handle data path errors. Early adopters must stay up to date with the most recent component firmware versions to ensure access to the latest capabilities as the standard evolves. They must also thoroughly test components in their specific workload environments prior to shifting into production.
End-to-end NVMe over FC is on the horizon. FC-NVMe gives IT vendors rules to use to resolve any disconnects that arise during interoperability testing. With the rules in place, it's expected vendors will roll out products soon.
However, some vendors have looked at where FC-NVMe is headed and plan to deliver systems prior to the standard's completion. For example, prior to publication of the standard, NetApp announced an NVMe-over-Fabrics (NVMe-oF) product with Broadcom's Brocade and Emulex divisions. Vexata announced support for NVMe-oF on multiple transports, including FC.
There are also storage arrays available that are NVMe-oF-ready, including FC-NVMe arrays. Dell EMC launched PowerMax as NVMe-oF-ready. IBM's FlashSystem already supports NVMe over InfiniBand, with support for additional transports expected in the future. Pure Storage brought out its new FlashArray//X portfolio as end-to-end NVMe and NVMe-oF-ready. Kaminario has done the same with its K2.N and Western Digital's Tegile with its IntelliFlash N-Series.
And where support isn't official yet, efforts are still underway. For example, Hitachi Vantara offers NVMe in its hyper-converged Unified Compute Platform and sees NVMe opportunities across other parts of its storage product line.
FC-NVMe gives IT vendors rules to use to resolve any disconnects that arise during interoperability testing.
Pay attention to the individual components. Storage systems, switches and host bus adapters are important pieces, but there are other components to consider. The OS, for one, must support FC-NVMe as well. SUSE Linux does, but FC-NVMe is still just in the plans for other OSes.
Also, check multipathing software. The good news is NVMe storage vendors are touting performance advantages even when running in traditional FC SANs. As a result, some performance benefits can be realized while other components catch up. Early adopters should understand as FC-NVMe is added to an increasing numbers of existing devices, updates may be required to resolve any new hiccups that arise.
New infrastructure required?
No one wants to rip and replace infrastructure. Knowing this, FCIA designed FC-NVMe to fit into existing SAN fabric infrastructure -- no new hardware or infrastructure investment required. This is incredibly beneficial to current FC SAN environments, which are likely already NVMe-ready.
NVMe replaces the SCSI command in the FC payload. As such, FC-NVMe behaves similarly to SCSI over Fibre Channel. NVMe technology works with SAN zoning, and the FC Name Server within a fabric can identify NVMe-over-FC ports. This lets a single name server know all the ports on the fabric and the types of protocols supported, simplifying management of a mixed protocol environment.
Actual support may vary, of course. So check with your vendor. Also, even though older-generation storage and networking products may support FC-NVMe, the most recent generations likely benefit the most from the new standard. Broadcom, for instance, claims its Brocade Gen 6 switches feature software optimization to reduce latency and integrated network sensors that provide better insight into network health and the performance of NVMe traffic.
The future of the data center
NVMe over FC will most likely improve the performance of FC SAN environments. As a result, applications should receive performance improvements, and SANs will become more efficient, extending the life of existing infrastructure and probably improving TCO.
The big question is will the performance improvement be enough to serve workloads using internal NVMe SSDs? SAN architectures have historically boasted a number of benefits over internal server storage and DAS, such as improvements to manageability, data protection and infrastructure efficiency. But an emerging set of workloads that demand ultra-low latency, such as high-velocity analytics and AI and machine learning, have found a home on DAS with NVMe SSDs. Can FC-NVMe bring these ultra-low-latency workloads to an FC SAN?
FC-NVMe SAN is expected to add only 10 microseconds of latency, but that isn't the whole story. The answer will likely be determined by where the bottleneck is in the data path. When flash replaced spinning hard drives, the performance bottleneck shifted away from the storage media to the network. NVMe over FC is supposed to fix that. But where will the new bottleneck be? It depends.
It could move to the application. Then FC-NVMe SANs will have a chance to consolidate these ultra-low-latency workloads. If it moves to the storage controller, however, then organizations may still need to support both architectures, with ultra-low-latency workloads remaining on DAS. In this scenario, improving NVMe efficiency within the storage system architecture may become an even more important differentiator for storage vendors. The extent to which FC-NVMe reduces latencies in FC SANs is definitely an area to watch in the coming months.
