Zerto keeps Texas bank from being left in the cold

When Winter Storm Uri cut off its access to grid power, Woodforest National Bank had to enact an emergency data center failover with Zerto.

The massive snowstorm in mid-February created rolling blackouts that shorted out the Houston-based community bank's automatic transfer switch, preventing it from switching over to generator power. This left Woodforest's data center with only UPS battery power, which amounted to about 30 minutes when the issue was discovered.

Luckily, Woodforest has a secondary data center at a colocation facility in Austin, Texas, and immediately began migrating servers over to it. However, at around the 20-minute mark, the Houston building lost power completely. At that point, Woodforest enacted its disaster recovery (DR) plan and used Zerto to fail over the remaining servers to the Austin site.

We had limited people, limited internet, limited power.

"We had limited people, limited internet, limited power," said Marcus Lohr, Woodforest National Bank's IT infrastructure manager.

Even under the circumstances, all customer-facing functions, including credit card processing and the website portal, were available within 90 minutes. Within four hours, Woodforest was back to running at full capacity as applications and ancillary business functions were brought back online.

Woodforest's infrastructure was fully tested and prepared for natural disasters, Lohr said. Using Zerto's replication technology, the company had been performing planned migrations between its Houston and Austin sites for the past 12 years.

This two-site setup was initially in response to hurricane season, Lohr said. From July to January, Woodforest would run its servers out of the hardened colocation facility in Austin, and for the rest of the year, the Houston data center would be primary. The two locations alternated between primary and secondary roles every six months, and both locations ran Cisco Unified Computing System on the same hardware.

Marcus Lohr

Zerto serves as Woodforest's primary data protection product for rapid and granular recovery, but it can typically recover only data that's three to seven days old. The bank uses a different backup product for retaining and recovering data that's older than seven days.

Woodforest had refined its DR plan to the point that migrations and failovers have gone from taking a day to down to about an hour, Lohr said. Outside of the technological capabilities provided by Zerto, Woodforest had also worked out a pecking order of who to contact, and all its DR procedures are hosted in a wiki that is accessible even if Woodforest's main site is down.

Additionally, the February outage took place just two weeks after a planned migration, so migration was still relatively fresh on staff members' minds, Lohr said. Despite the fact that only about 25% of the people responsible for DR were available during the winter storm incident, everyone remained calm and knew how to carry out the transition, according to Lohr.

"Twelve years of doing this and trying to make it better, the business has done such a good job with it and nobody lost their heads during this," he said.

In order to prepare for the next big outage, Lohr has started investigating another colocation facility. He said he plans to move all data center activity out of Woodforest's main building, such that both primary and secondary data centers will be in hardened facilities with redundant power, cooling and security year-round.

About 90% of Woodforest's infrastructure is virtualized, but the next step will be to involve cloud more, Lohr said. Woodforest currently has very little cloud infrastructure, with its largest investment being email in the cloud through Office 365.

There are many benefits to the cloud, such as being able to expand Woodforest's DR options, but it's something he needs to approach carefully to ensure he's optimizing costs, Lohr said.

"We're dipping our toes in the cloud right now, but we've been moving towards it. We see it as a tertiary data center in the future," Lohr said.