The future of storage is disaggregated, however you cut it

Disaggregated storage vs. HCI

Disaggregated storage is an alternative to HCI for a broad range of standard business applications. HCI has become popular over the last decade as a simpler way to deploy and manage infrastructure for a range of core business applications.

Instead of managing separate storage and compute environments -- historically a complex undertaking -- HCI vendors developed clever software that allowed VM admins to manage their virtualized applications running on special configurations of commodity server hardware that also includes shared, highly available and performant storage.

The benefits of this approach in terms of simplified management and operations can be profound -- hence the surge in HCI adoption over recent years. But there can be drawbacks. For example, the scaling model of HCI is more rigid; this can lead to resource underutilization if, for example, storage requirements vastly outpace compute needs.

Though there are workarounds to this scaling challenge, the other major driver toward disaggregation comes courtesy of changes in the virtualization landscape. VMware's acquisition by Broadcom has seen strategic shifts in the industry alliances that play a critical role in driving enterprise technology adoption.

For example, Dell -- which, as the previous owner of VMware, was instrumental in driving HCI adoption through joint offerings such as Dell EMC VxRail -- is in the process of changing its focus.

"You're seeing a pivot to Dell IP storage ... we think a disaggregated architecture is the right answer with modern workloads," Dell COO Jeff Clarke said during the company's most recent earnings call, along with a clear expectation that the company's HCI business will decline.

Meanwhile, HPE has been on the disaggregated storage path for some time and continues to position its combination of separate storage and servers as a "best of both worlds" capability that offers the scale and flexibility of HCI twinned with a seamless management experience associated with HCI, packaged as part of HPE's GreenLake private cloud capability.

While suppliers will clearly benefit by selling more storage based on their own IP, the challenge will be to convince customers that "disaggregated" storage offers significant benefits relative to HCI. Many of these customers will have switched to HCI from traditional three-tier architectures in the first place, so the incremental value that a "disaggregated" approach offers must be compelling.

As well as the scaling benefits of the disaggregated model -- where compute and storage can be added independently -- you can expect to see a big focus on improving the manageability of the overall environment; removing the complexity that historically was associated with three-tier architectures, providing greater flexibility and agility, offering flexible consumption models, and of course, lowering TCO. Ultimately, customers should stand to benefit from the greater focus on innovation here.

Whither HCI?

An obvious question is the extent to which these shifts will affect the HCI market overall. After all, Dell refocusing its attention away from HCI is bound to have some impact on overall adoption. However, I strongly believe HCI still has a solid future ahead of it. For a start, VMware remains strongly focused on promoting its own HCI -- vSAN -- and others, especially Nutanix, but also a group of smaller specialists, such as StorMagic and Verge.io, continue to make good progress. Most application environments stand to benefit from virtualized infrastructure, and the need to provide simplified management only increases as applications increasingly take advantage of cloud-native technologies such as Kubernetes.

Additionally, Dell, HPE, Lenovo and others will continue to invest in HCI for customers who prefer this approach. Indeed, Dell, ever the pragmatist who will strive to meet customers wherever they prefer, is working with Nutanix on a number of fronts to develop and promote joint offerings.

AI use for disaggregated storage

There's another area where you can expect to see a huge focus on disaggregated storage: the market for large-scale, high-performance unstructured data storage. This space is in sharp focus right now as organizations -- particularly those looking to deploy AI at scale -- grapple with the profound infrastructure challenges that this presents.

Running AI workloads such as large language model training and inference at scale will require the ability to process a vast amount of unstructured data at unprecedented speeds. Doing this using historic storage approaches is challenging; compute and storage resources within a storage architecture are typically tightly coupled, which can make scaling both incredibly difficult and incredibly inefficient if resource requirements grow at different rates.

This is much more likely in AI, especially when processing large amounts of unstructured data for "multimodal" generative AI workloads that might include combinations of text, images and video data. In addition to the regular storage processing for data management and protection that every storage system requires, more data processing is necessary for many AI workloads because unstructured data in its raw form is unusable; it needs to be transformed into a structure that the AI app can understand. This "translation" is performed using a metadata engine, which adds context and meaning to unstructured data in a way an AI app can use. Because of data gravity, proponents argue the most logical place for the metadata engine to run is within the storage environment.

Any AI-oriented storage architecture, then, needs to be able to effectively manage this combination of storage tasks, metadata and related tasks, and  back-end capacity. In an AI workload all three elements might grow at wildly different rates. The challenge is how to deliver all this cost-effectively and without requiring an army of storage experts to manage. These aspects get to the crux of why disaggregated storage will be integral to delivering AI at scale.

And that is not all. As AI workloads begin to penetrate the large enterprise space, organizations will require resilient solutions that allow them to maintain control of their data management, security, governance and compliance environments.

The notion of disaggregated storage in this space is already gaining traction, but this is set to ramp up significantly over the coming months. Newer storage suppliers such as Vast Data, with its Disaggregated Shared-Everything Architecture, were among the pioneers but are now being joined by the larger storage incumbents, all keen to get a slice of the AI storage pie.

For example, NetApp recently said it was developing a disaggregated storage variant of its core OnTap platform that will allow it to execute on its AI vision. This capability, it said, will enable full sharing of the storage back end, maximizing network and flash storage speeds to drive significant performance improvements, while also economizing rack space and power.

On March 11, Pure Storage launched FlashBlade//EXA, a variant of its FlashBlade architecture that utilizes a disaggregated architecture to independently scale data and metadata, in the process driving significant performance for AI and HPC environments. Similar to NetApp's initiative, FlashBlade//EXA has no specific delivery date but is expected to become available this summer. By then, we are also likely to have heard more details around Dell's initiative -- dubbed Project Lightning -- in this space.

In summary, we can expect to hear a lot more about disaggregated storage over the coming months. When considering a disaggregated storage system, it's important to focus on the type of disaggregation being offered. Whether for AI-centric workloads or more traditional business applications, there's potentially a variant that will work for your organization.

As ever, there is potential for confusion as the various players jostle for position, and efforts will need to continue to educate potential buyers of the relative merits of their offerings. But, in the ebb and flow of today's technology ecosystem, there's little doubt that disaggregated storage is going to see its fair share of the limelight in the months ahead.

Simon Robinson is principal analyst covering infrastructure at Enterprise Strategy Group, now part of Omdia.

Enterprise Strategy Group is part of Omdia. Its analysts have business relationships with technology vendors.