The challenge of managing modern enterprise storage needs will grow in the coming years, and movements across vendors will also grow more costly.
Hybrid on-prem and cloud storage deployments compose the storage infrastructure of more than 50% of customer organizations, according to numerous storage surveys by vendors and research firms. Hybrid deployments can solve a variety of enterprise needs, such as keeping frequently used data cached and quickly available while moving older, colder data into the cloud. But, as many of these recent surveys note, managing unstructured data across such an environment can escalate in cost and complexity quickly.
A 2020 survey of 1,500 enterprise storage customers by research firm IDC found that 55% of respondents make use of a hybrid cloud environment. The specifics of what IDC considers hybrid cloud storage includes at least some on-prem storage, alongside the use of public or private cloud compute and storage. Hybrid architecture can also vary from just a single cloud provider to multiple clouds, according to IDC's definition.
Most organizations, even if they back up to multiple clouds, realistically make use of the services of just one vendor, said Andrew Smith, a research manager at IDC.
"They might have multiple providers, but their strategy is primarily built around a single provider," he said. "The multi-cloud utopian dream is that enterprises would deploy applications in any public cloud … but that dream is way far off from reality."
The report further noted that 30% of organizations intend to make use of a hybrid cloud, and 16% have no interest in using cloud services.
Many IT enterprise users get direction from executives to utilize multiple clouds to avoid vendor lock-in for services, but they end up using just one primary cloud for a majority of their storage and compute, according to Smith.
"When we ask enterprises why they use multiple cloud providers, the No. 2 reason they have is active guidance from their enterprise to try and mitigate lock-in," he said. "In theory, that's good, but in practice, I'm not sure."
Staying agile with enterprise data and applications is the dream. The manpower and time it would cost to rewrite and redirect applications and storage between platforms leads most IT departments to stick with one choice of cloud, Smith said.
Major public clouds are attempting to make inroads into customers' on-prem data centers through services such as AWS Outposts or Azure Stack. Their exact impact on locking enterprises to just one major cloud remains to be seen, however, he added.
"It's kind of the Wild West over who is going to win the hybrid cloud race," Smith said. "The No. 1 factor for most enterprises is cost. A lot of these data management providers help mitigate those costs."
Data middle management
Surveys conducted this year by data management vendors Komprise and Next Pathway echoed IDC's report, finding hybrid deployments becoming common across organizations.
Storage consultants agree that data management services can help maintain insight and control over hybrid environments better -- particularly for disaster recovery -- but cautioned against seeing such purchases as a silver bullet for escalating costs and manpower needs.
According to Komprise's "State of Unstructured Data Management Report" from August, enterprises are managing between 1 to 5 petabytes of data, and these enterprises expect to spend about 30% to 40% of their IT budget on data storage and protection.
The Komprise survey featured respondents from 300 enterprises across the globe, each with more than 1,000 employees.
"There were no surprises. It was reconfirming all the things we were hearing from customers," Komprise COO Krishna Subramanian said.
Storage experts similarly expect data, especially unstructured data, to continue to grow. However, data management tools are only one part of an architecture solution and can become another costly addition or vendor lock-in to work around.
"Data is exploding," said Ray Lucchesi, president of Silverton Consulting. "It's where we've been for the last decade. The numbers are getting extreme. Unstructured data has no end in sight."
The compute and AI capabilities of the cloud mean organizations will inevitably develop in the cloud, resulting in an ever-shifting target for budgets to account for as users and storage needs scale up and down.
"The reality is for IT, there's a good case to run stuff in the cloud and there's a good case to run on-prem," Lucchesi said. "If it's not running there now, you have to move the data."
You're not reducing your cost. You're reducing the cost for a separate data center.
Marc StaimerPresident, Dragon Slayer Consulting
Marc Staimer, president of Dragon Slayer Consulting, said enterprises should always consider investments and purchases for the cloud akin to any other IT architecture purchase.
"You're not reducing your cost," he said. "You're reducing the cost for a separate data center."
Echoing Lucchesi, Staimer agreed that hybrid deployments can quickly add up in architectural complexity and overall cost as more employees make use of an organization's cloud applications and storage.
"You're not reducing cost, you're increasing it," Staimer said.
He suggested organizations considering data management services should make sure vendors allow reading and writing to data without needing their specific storage software. He also suggested organizations should consider any vendor that doesn't charge data egress fees, as the inevitable need to change can lead to a massive surge in costs.
He noted that Komprise, Strongbox, Datadobi and HammerSpace all handle data management for hybrid environments but are effective only at certain data sizes. Strongbox is designed for frequent management of petabytes to exabytes of data over competitors, he said.
Although particularly scrappy IT departments may consider using open source or freely available applications and tools, like Rsync or Robocopy, but Staimer cautioned such tools lack the error detection and security features available from vendors.
Lucchesi said he considers the data growth an optimistic point for storage professionals and the industry, since demand is just going to increase for skills and services.
"It's a great place to be in the storage industry. It never stops," he said.
