Are you ready to get started with NVMe over fabrics to extend NVMe speed across the storage network? Then, the first thing to understand is that NVMe-oF is designed to use your existing storage fabric. The NVMe over fabrics standard allows the use of Fibre Channel, InfiniBand, iWARP, RoCE and TCP. However, because of the protocol's binding requirements, being able to use the technology over your fabric of choice requires vendor support.
The first step in implementing NVMe-oF is typically to verify its compatibility with your existing storage infrastructure. As you do this, it's important to keep in mind that NVMe over fabrics is still relatively new, and not every vendor rigidly adheres to the NVMe-oF standard. If a vendor has its own way of doing things, it can lead to compatibility problems. As such, it is a good idea to make sure that your vendor isn't pushing its own proprietary approach.
If this is your first experience with NVMe over fabrics, then your initial implementation should be in a lab environment. Although the technology is well-suited for use with mission-critical enterprise workloads, it's wise to get a feel for the technology before bringing it into a production environment.
Relatively simple deployment
Deploying NVMe over fabrics tends to be straightforward. Although each vendor does things a bit differently, the process of setting up the technology is similar to setting up iSCSI connectivity. At a high level, the process involves creating an NVMe-oF target on your SAN or virtual SAN and then configuring an initiator to connect to that target.
How NVMe-oF works
In spite of the relative simplicity of the deployment process, there are three important areas to consider as you get started.
1. Do you have adequate throughput?
The primary appeal of NVMe storage is its speed. But that means the connectivity between the initiator and the target must be fast enough to keep up with the physical NVMe storage device. Otherwise, there won't be any real benefit to implementing NVMe over fabrics.
Although NVMe-oF is well-suited for use with mission-critical enterprise workloads, it's wise to get a feel for the technology before bringing it into a production environment.
Current-generation enterprise-class NVMe storage devices have a throughput of 32 GBps. As such, a 40 GBps connection between the initiator and the target should suffice -- but with a couple of major caveats.
The first caveat is the 40 GBps connection must be dedicated for use only by the target and the initiator. Otherwise, competing traffic could easily deplete the available bandwidth.
The second caveat is the 40 GBps connection should only be used if the target is tied to a single physical NVMe device. If multiple NVMe storage devices are striped together, then you must use a faster connection between the initiator and the target. One popular choice is the Mellanox ConnectX-6 adapter, which features twin 200 GBps ports. Mellanox also offers a ConnectX-5 adapter with twin 100 GBps ports.
2. Does your adapter support RDMA?
Another factor to keep in mind is that NVMe drives connect through the PCIe bus and access the system's memory through remote direct memory access (RDMA). The adapters that provide the physical connectivity between the target and the initiator should support RDMA and zero-copy networking. Without this, the connectivity may be subject to CPU latency or may not work at all.
Fabrics designed for use with NVMe-oF, such as InfiniBand, Internet Wide Area RDMA Protocol (iWARP) and RDMA over Converged Ethernet (RoCE), natively support RDMA. In fact, nearly every high-performance adapter supports RDMA, but it's still a good idea to confirm that your adapter meets this requirement.
3. Are your device drivers and OS kernels up to date?
Finally, it's extremely important to make sure your software is up to date. Problems getting NVMe-oF to work properly often stem from outdated device drivers and OS kernels. It's also worth noting that vendors usually require the NVMe storage device to be provisioned prior to mapping a target to the disk. For example, a vendor might require the disk has been initialized as GPT, or GUID Partition Table, and a NTFS (New Technology File System) partition has been created on the disk. The actual requirements vary by vendor.
Setting up NVMe-oF for the first time isn't a monumental challenge. The key is to verify the vendor's requirements upfront and ensure that all of those requirements have been met prior to trying to establish connectivity.
