State tax agency nixes slow HDD backup for speedier flash
The Mississippi Department of Revenue chose an all-flash storage and backup setup to maximize app performance and minimize headaches after a spinning-platter disaster.
The Mississippi Department of Revenue turned to flash storage for its DevOps teams following a hard disk backup appliance failure and found it to be a similarly speedy and reliable medium for handling its backups.
The DOR, headquartered in Canton, Miss., manages a fluctuating amount of data between 100 to 200 TB in its two data centers as well as local storage on edge points such as faxes and scanners, according to Mike Dehaan, enterprise architect at the DOR.
Dehaan oversees the DOR's infrastructure engineering team, which supervises IT systems across the network. Dehaan acts as a resource to team members and reports to the CIO.
The DOR's total storage also includes about 12 TB of data backups managed daily, which the DevOps team uses as a testing ground before moving codebase changes into the production environment.
The backup process involved a backup appliance which, although cost effective, proved untenable.
"We did the thing that every storage admin has done for a very long time: backups, cheap and deep," Dehaan said. "Let's just get all the spinning rust we can find and just throw it out there."
Pure uses only flash memory, a more expensive medium compared to the hard drives they had been using. But Dehaan said the performance and stability of flash ensured his team could not only depend on its backups but also improve overall productivity.
"We learned just how deeply our applications depended on our storage [infrastructure] for software development," he said. "It was a legitimate problem for our piece of business. We just weren't able to effectively utilize our development resources at that point."
Change in a flash
The DOR has two major applications and associated databases: Marvin, the Mississippi Automated Registration Vehicle Information Network; and Mars, the Mississippi Automated Revenue System. The department serves a total of 82 counties and a population of almost 3 million.
Dehaan said the backup appliance prior to FlashBlade suffered degrading performance as the number of storage clusters increased over time. Database backups took upwards of 24 hours and, occasionally, more than an entire weekend to backup. Dehaan declined to provide the vendor or how long the DOR had been using the appliance.
Although the slow performance was workable, an infrastructure-wide storage system failure one weekend brought the situation to a head.
"We had a corrupted database -- backups we could not restore from. So, we began walking down the path of manually restoring," he said. "If I ever have to go through a situation like that again, I might just call it quits on IT and just go start [ditch] digging."
If I ever have to go through a situation like that again, I might just call it quits on IT and just go start [ditch] digging.
Mike DehaanEnterprise architect, Mississippi Department of Revenue
An exact reason for the failure couldn't be diagnosed by the DOR's three storage infrastructure vendors, which included Pure Storage, Microsoft and VMware.
Rather than putting together another backup appliance with hard disks and running into undiagnosable issues once more, Dehaan and his team decided to try FlashBlade based on their experiences with FlashStack.
"We already tried a budget-friendly solution with spinning disks, and we were thoroughly disappointed," he said. "We had already gotten the gnashing of teeth out of the way trying to get performance out of a lesser platform. It was time to meet our needs, and we allocated the appropriate funds."
Dehaan said the improvements to backup performance and speed by using an all-flash array were immediate.
"We went from 18- [to] 24-hour restore windows to three or four hours," he said. "We were able to tangibly see the pace of development increase right out of the gate."
Initial performance tuning and setup required just in-house staff and resources, Dehaan added.
"It took very little from the storage team to get it tuned up," he said. "Most off the tweaks were on the database side. Previously, storage was the bottleneck."
Spend money to make money
Dehaan said his team purchased FlashBlade using Pure's Evergreen//Forever buying option, which gives the customer ownership of the hardware with a subscription to hardware refreshes, Pure's software and support services.
Deehan acknowledged the team initially had sticker shock but found that ongoing hardware refreshes work better with government budgeting cycles and limitations. Evergreen//Forever gives the department a fixed cost to work with rather than needing to restart the process to procure new hardware every few years, he said.
"We can space that out over the years [as] a recurring yearly cost and that works so much easier," Dehaan said. "The initial cost of storage arrays from Pure does come at a premium, as any all-flash array does. But that is offset by the lifecycle of the storage. To us, Evergreen is just a mechanism to purchase a new array over the lifecycle between refreshes and not have to do a rack-and-stack [or] migration when the refresh time is due."
He also said Pure's hardware support and documentation made operations easy for him to follow and only required a junior engineer with the DOR to monitor updates.
Tim McCarthy is a journalist living in the North Shore of Massachusetts. He covers cloud and data storage news.
Mike Dehaan
