Part of:Scale storage to deal with increasing data volumes
How to plan for a potential data storage shortage
Try saying 'data storage shortage' five times fast. Hopefully you won't need to say it often in the near future, but with escalating data volumes, planning is key.
With the skyrocketing rate of data growth, it's a wonder how data centers, enterprises and people can store so much information.
Is the storage industry likely to reach a point when a "No Vacancy" sign will hang, telling users that they will need to discard all new data as soon as they generate it? Will users have to discard their history to make room for new data? Or will IT find satisfactory homes for this data deluge? The answer isn't simple.
How limited is the storage industry?
In a 2018 study, IDC predicted that by 2025 there will be 175 zettabytes of data that needs storage. Speakers at storage industry trade shows, including the Flash Memory Summit and Storage Networking Industry Association events, like to share this seemingly overwhelming extrapolation of future data requirements. The 175-zettabyte figure, while it appears large, represents an average 27% growth rate between 2018 and 2025. However, the figure also demonstrates the potential need for a large amount of storage space within just a few years.
Today, both flash memory and HDD companies exercise discipline in their production growth. Although they could ship more, they find that it's more profitable to remain just shy of creating an oversupply, since that would trigger a price collapse. Flash makers are currently managing their output to grow at a rate of about 30%, while HDD makers are increasing their shipments at an annual rate of about 20%, according to Objective Analysis evaluation.
Vendors review and trim these growth figures every quarter, and judiciously add manufacturing capacity to support them. As a result, it's unlikely that any inadvertent oversupply before 2025 would continue for long.
NAND flash shipments from SSD makers are still small compared to HDD makers' shipments, and will likely remain far behind for a long time. Expect HDDs to store the majority of the 175 zettabytes.
However, since the HDD business is growing its shipments at a slower rate than IDC's 27% projection, is the industry setting itself up for a data storage shortage?
Perhaps, but keep in mind that the use of data compression is on the rise, and that means that those 175 zettabytes won't look as large when 2025 finally arrives. While HDD makers are likely to fall short of the need for capacity, the data storage shortage shouldn't be apocalyptic in the next five years. Over the longer term, new technologies like DNA storage will emerge and provide devices that store the equivalent of today's entire internet in a space no larger than only one or two equipment racks.
Still, anticipate a shortage to be in a better position to handle it.
Anticipate a data storage shortage to be in a better position to handle it.
Implications of a data storage shortage
How will this impact the way that businesses work over the next few years? Here are a few scenarios:
Even though the manufacturing cost of storage bytes will decrease, shortages tend to keep market prices from decreasing as quickly as they should. If the shortage is bad enough, the price of storage could even temporarily rise, as flash memory prices did in 2017.
An actual shortage could drive challenges for storage admins to solve: Should they archive fewer backups? Should they vet data better before storing? Should they retire old data at an earlier date? These questions will drive corporate-level review and will likely result in policy changes.
In certain cases, storage architecture will need to adapt to the developing data storage shortage before it has an opportunity to evolve into a crisis. Organizations should seriously analyze their data center structures to reveal areas of duplication and opportunities to take advantage of compression or better data management.
What can storage admins do?
Most importantly, storage admins should align their thinking and planning around the idea that storage will become less abundant. It's time to rough out a plan and begin to talk about it, not only within the IT department, but also with the various parts of the company that will need to respond in their own way, from IT end users to those who sign the checks for capital spending.
Admins can use a three-pronged strategy.
End users. End users will unwittingly create storage issues. It's helpful to discuss prospective issues with them and educate about what is and is not important. For example, they may save duplicate copies of data in multiple places to make it easier to find. There may be obsolete files that the organization can discard. In these cases, the user or programmer didn't realize they should exercise some discipline and instead squandered storage space. With a little encouragement and guidance, these same users and programmers should include storage considerations into their plans and routines.
Advanced tools. Take the time to learn all the options in some detail. Understand the latest trends in storage management and the ins and outs of compression. For example, admins can compress HTML to 17% of its original size. Proper data management can reduce the need for overprovisioning and reduce SSD wear. Gain a better understanding of how the organization uses and stores data. Explore ways to improve data storage efficiency.
Technology forecasts. Follow forecasts to learn where the industry is headed to help second-guess potential problems before they occur or avoid them altogether. Consider contingencies and plan around them. For example, plan for how an increase in storage costs might affect the capital budget. Get management involved at the highest possible level -- upper management detests being blindsided. They will appreciate the warning. The end-user communication will also help accurately plan for company requirements.
