Future of the Ethernet drive may hinge on NVMe over Ethernet
Despite their debut two years ago, drives that connect via Ethernet have been mostly a curiosity. The Ethernet flavor of NVMe over Fabrics may change that.
Ethernet disk drives were rolled out by Seagate Technology LLC in 2014 with quite a bit of hoopla. Western Digital (WD) Corp./HGST followed a few months later with its version. The two drives are different subclasses of Ethernet drives, and are not even close to compatible. The Seagate Kinetic HDD uses a key/data addressing scheme clearly aimed at big data, while the WD product has a full Linux server running on the drive.
Two years later, we are still waiting for further major announcements. Several vendors, including Supermicro, support the approach with a JBOD chassis capable of supporting Ethernet drive connectivity, rather than SAS/SATA. In addition, Seagate is keeping technology current with their September 2015 announcement of an 8 TB version of the Kinetic drive.
Still, there have been no major announcements about the Kinetic drive in more than a year, and WD placed its efforts around Ethernet drives on the back burner for most of 2015 and 2016. The vendor simply couldn't identify compelling use cases.
Part of the problem is that drive makers understand, well, drives. They tend to leave systematization to other companies and open source communities.
For example, let's take Kinetic. It has a REST interface, which fits the object storage market, but object stores don't talk REST to drives. They access appliances with REST, and the appliances hide the structure of multiple drives and complex redundancy systems that a typical object store is built upon. Simply put, Kinetic doesn't fit the most common object store, Ceph, or any of the other packages around.
The WD approach lends itself to offerings that use more sophisticated software in the drive interface. For instance, iSCSI or Ceph could be used to connect the unit, but the issue of system data distribution, as opposed to drive storage, which the Kinetic drive ran into, also gets in the way of a useful deployment of Linux-based Ethernet drives.
WD's research unit, WDLabs, made inroads on the Ethernet question when it recently built an experimental 504-drive Ceph cluster, which runs the Ceph object storage daemon code on each drive. This was not one of the production Ethernet drive types. Rather, the lab cobbled together a design using a separate ARM64 server CPU. What was most telling about this design was that it had dual 10 GbE ports, which looked like a major nod to Ceph bottlenecks, and which may have pointed to one of the reasons for the slow start of Ethernet drives in general.
The structure of the Storage Network Industry Association (SNIA) Ethernet Drive Management Spec (which is currently in draft form) reflects the paucity of use cases and of a concerted marketing effort. It supports not only the two announced classes of products, but four more classes that look more like the higher level functionality that is needed for useful deployment. However, this still looks like a spec searching for its market.
Hope on the horizon
All is not lost on the Ethernet drive side. One of SNIA's hottest projects is defining Nonvolatile Memory Express over Ethernet. This is most definitely a case of a spec trying to catch up with development.
There are use cases galore for a fast remote direct memory access (RDMA) long-haul interface for flash/SSDs. All-flash array vendors are driving hard for 40 GbE with RDMA, and will move to 100 GbE/RDMA later this year as it becomes available.
NVMe is a block/IO interface, but it has the potential to carry any storage access mode, including file and object traffic, and even byte-addressable access (think nonvolatile dual in-line memory module). It fits faster flash and SSD appliances perfectly, with low overhead in the processors, zero-copy transfers and low latency. By any assessment, NVMe over Ethernet is set to replace the Fibre Channel and bring an Ethernet SAN/LAN convergence into cloud-based clusters.
It is almost certain we'll evolve to a mechanism where the drives connect to an Ethernet switch, allowing direct access to the virtual SAN.
Will NVMe change the fortunes of Ethernet as a universal drive fabric? That's a complex question because it involves vendors' pricing tiers and the existence of expensive enterprise drives. The answer to the question is more likely to lie with the move from the traditional servers connecting storage arrays model of systems to the storage appliances also running containers with apps model that hyper-converged and software-defined infrastructures bring to the table.
Hyper-converged infrastructures need a virtual SAN to share storage across the cluster. The network of choice is Ethernet/RDMA, with internal drives in each server/appliance connected to Ethernet for sharing. In our unevolved products today, this is done by bringing data from the drives into the CPU and then sending the CPU out to an Ethernet network interface card. Even with RDMA, this slows transfers down a great deal and adds a lot of overhead.
It is almost certain we'll evolve to a mechanism where the drives connect to an Ethernet switch, allowing direct access to the virtual SAN. Should that happen, there will be a use case for making slower bulk SSDs (hard drives belonging to history within a few years) also talk NVMe over Ethernet.
This could be the route home for Ethernet interfaces on drives. It certainly simplifies life if we unify the interfaces on Ethernet, and it will speed up deployment of faster links in storage and avoid the need for a separate SAN management team.
Next Steps
Red Hat exec talks about uses for Ceph
Ceph, Swift and other object alternatives for storage
