Is an all-flash data center worth it?

Hurdles of the all-flash data center

In reality, asterisks associated with the price parity claim make creating a single-flash tier difficult. First, the price comparison is being made to high-performance hard disk systems, not more cost-effective high-capacity systems. Second, most all-flash vendors design their arrays for block-based data. And while some incorporate NAS functionality, a few have also integrated object storage, which is quickly becoming the storage method of choice for unstructured, machine-generated data. There are some exceptions, but generally, most unstructured data use cases simply can't take advantage of flash performance. Flash is almost always a perfect match for block storage, but not always the ideal storage type for NAS storage or objects. Hard disk drives are often the better, more cost-effective choice. Third, even within the flash category, there are multiple types of flash media, each better for certain use cases (see "Capacity-centric all-flash").

Then there is the data. Even though organizations create and store more data than ever, they will never access most of that data again after the initial creation and modification phase. But, of course, we keep that data "just in case." If there weren't cheaper alternatives, then storing all this data on flash would be fine. So when just in case occurs, the response would be instant.

The problem is, at least for all-flash data center proponents, there are less expensive storage options available, three to be exact: tape, high-capacity HDDs and the cloud. Each of these has its negatives: Tape is operationally expensive, cloud introduces new security considerations and all have administrative challenges with transferring data from an all-flash tier. But there is little doubt moving older data to one of these older and slower technology tiers should save money.

That brings us back to the original question, "How much flash is enough?" The answer -- as is almost always the case in IT -- is, it depends. Generally, use as little flash as possible, as long as you can find cost-compelling alternatives. To realize these savings, you will need a method to move old data from high-performance storage to cost-effective storage as seamlessly as possible, however. If transparent data movement isn't achievable, then the savings gained by implementing a cost-effective tier of storage will quickly be lost.

That means the deciding factor to the all-flash data center is the effectiveness of your data movement strategy.

Effective data movement strategies

The concept of moving inactive data from an expensive tier of storage to a less expensive tier has been around for decades. The problem is most data movement options were terrible, and IT professionals came to the conclusion that managing their data wasn't worth the effort. But things changed in recent years, and moving data between tiers of storage is now easier to implement and manage.

The first step making the movement of data easier was the introduction of hybrid storage systems. These storage systems move data within themselves, typically from a small flash tier to a large hard disk tier. Interestingly, most hybrid vendors now also provide all-flash products and indicate their all-flash arrays outsell their hybrid systems.

Some vendors point to their all-flash products outselling hybrid systems as proof that the move to the all-flash data center is well underway. While hybrid systems make sense for many environments, the appeal of all-flash is too alluring. The challenge for hybrid systems is they need to be capable of delivering all-flash performance and hard disk performance. Since the internals -- compute, memory, networking -- of a hybrid system must sustain the performance capabilities of flash, the price of those internals will be higher than if it were a standard array that only needed to sustain the performance capabilities of hard disk drives.

To an extent then, all-flash vendors are correct in stating that for high-performance, active or near-active data, a couple of flash systems may be ideal. The future of hybrid systems (see "The role of hybrid arrays") is in managing different types of flash, not in managing flash and HDDs.

There is still inactive data to contend with, but there must be a more cost-effective way to store it. A storage system solely focused on that data type enjoys significant cost-per-gigabyte advantage over hybrid arrays, but it introduces a separate system. Data centers need to move data between these different types of storage systems on a policy-driven basis. The good news is there are a growing number of software products to do just that.

Hardware-independent data movement

Hardware-independent data movement products come in several forms. The first is a global file system. These are file systems that stretch across physical storage hardware components and move data between them based on policy. The types of hardware that can be part of a global file system is almost unlimited, with most supporting movement between flash and hard disk-based systems, some supporting movement to cloud storage (private and public) and some even supporting movement to tape. Given that global file systems are enhanced file systems, they limit support to data that can reside on NFS or SMB mounts. All the storage hardware part of a global file system must also be able to create an NFS or SMB share for it to manage.

In addition to global file systems, there's a new type of hardware-independent data movement from companies such as Primary Data, ioFabric and Strongbox. Their software does not require all data be hosted on a file share or all participating storage hardware to present its storage as an NFS or SMB share. Some of these can even encompass block or object-based storage systems and move data between them.

In either case, hardware-independent data movement allows you to implement all-flash arrays to increase performance of active data and less expensive, capacity-optimized storage systems for inactive data. It transparently moves data between these task-specific systems based on policies established by the organization, the most common of which would be "If data has not been accessed in a set number of days, move it from the flash arrays to capacity storage."

So while all-flash arrays are at price parity with hard disk-based arrays for active data, there are still less expensive alternatives for inactive data. High capacity HDD arrays, object storage, public cloud storage and even tape perform adequately at a lower price than flash arrays. The case for an all-flash data center is that the effort involved in deciding what data should reside where, and how to move that data to the alternate storage types can be more costly than the upcharge for flash everywhere. The problem is this stance ignores just how inexpensive capacity storage products are and how easy it is becoming to seamlessly move data to them.

Most data centers should only consider all-flash arrays for active and near-active data, at most 20% of total capacity. The remaining 80% of data center storage should reside on capacity storage -- high-capacity NAS, object storage or public cloud storage.

So  maybe we should be talking about one-fifth flash data center instead of an all-flash data center.