Database DevOps tools bring stateful apps up to modern speed
Database DevOps is no longer a contradiction in terms, as enterprises roll out database management and persistent storage software that makes stateful apps more flexible.
DevOps shops can say goodbye to a major roadblock in rapid application development.
At this time in 2017, cultural backlash from database administrators (DBAs) and a lack of mature database DevOps tools made stateful applications a hindrance to the rapid, iterative changes made by Agile enterprise developers. But, now, enterprises have found both application and infrastructure tools that align databases with fast-moving DevOps pipelines.
"When the marketing department would make strategy changes, our databases couldn't keep up," said Matthew Haigh, data architect for U.K.-based babywear retailer Mamas & Papas. "If we got a marketing initiative Thursday evening, on Monday morning, they'd want to know the results. And we struggled to make changes that fast."
Haigh's team, which manages a Microsoft Power BI data warehouse for the company, has realigned itself around database DevOps tools from Redgate since 2017. The DBA team now refers to itself as the "DataOps" team, and it uses Microsoft's Visual Studio Team Services to make as many as 15 to 20 daily changes to the retailer's data warehouse during business hours.
Redgate's SQL Monitor was the catalyst to improve collaboration between the company's developers and DBAs. Haigh gave developers access to the monitoring tool interface and alerts through a Slack channel, so they could immediately see the effect of application changes on the data warehouse. They also use Redgate's SQL Clone tool to spin up test databases themselves, as needed.
"There's a major question when you're starting DevOps: Do you try to change the culture first, or put tools in and hope change happens?" Haigh said. "In our case, the tools have prompted cultural change -- not just for our DataOps team and dev teams, but also IT support."
Database DevOps tools sync schemas
Redgate's SQL Toolbelt suite is one of several tools enterprises can use to make rapid changes to database schemas while preserving data integrity. Redgate focuses on Microsoft SQL Server, while other vendors, such as Datical (now Liquibase) and DBmaestro, support a variety of databases, such as Oracle and MySQL. All of these tools track changes to database schemas from application updates and apply those changes more rapidly than traditional database management tools. They also integrate with CI/CD pipelines for automated database updates.
Editor's note: Datical rebranded in 2020, as is now called Liquibase (a commercial version of the open source software).
Radial Inc., an e-commerce company based in King of Prussia, Pa., and spun out of eBay in 2016, took a little more than two years to establish database DevOps processes with tools from Datical. In that time, the company has trimmed its app development processes that involve Oracle, SQL Server, MySQL and Sybase databases from days down to two or three hours.
"Our legacy apps, at one point, were deployed every two to three months, but we now have 30 to 40 microservices deployed in two-week sprints," said Devon Siegfried, database architect for Radial. "Each of our microservices has a single purpose and its own data store with its own schema."
That means Radial, a 7,000-employee multinational company, manages about 300 Oracle databases and about 130 instances of SQL Server. The largest database change log it's processed through Datical's tool involved more than 1,300 discrete changes.
"We liked Datical's support for managing at the discrete-change level and forecasting the impact of changes before deployment," Siegfried said. "It also has a good rules engine to enforce security and compliance standards."
Datical's tool is integrated with the company's GoCD DevOps pipeline, but DBAs still manually kick off changes to databases in production. Siegfried said he hopes that will change in the next two months, when an update to Datical will allow it to detect finer-grained attributes of objects from legacy databases.
ING Bank Turkey looks to Datical competitor DBmaestro to link .NET developers who check in changes through Microsoft's Team Foundation Server 2018 to its 20 TB Oracle core banking database. Before its DBmaestro rollout in November 2017, those developers manually tracked schema and script changes through the development and test stages and ensured the right ones deployed to production. DBmaestro now handles those tasks automatically.
"Developers no longer have to create deployment scripts or understand changes preproduction, which was not a safe practice and required more effort," said Onder Altinkurt, IT product manager for ING Bank Turkey, based in Istanbul. "Now, we're able to make database changes roughly weekly, with 60 developers in 15 teams and 70 application development pipelines."
A screenshot of Datical's software shows database testing pipelines.
Consistent database schemas and deployment scripts through rapid application changes is an important part of DevOps practices with stateful applications, but there's another side to that coin -- infrastructure provisioning.
Stateful application management through containers and container orchestration tools such as Kubernetes is still in its early stages, but persistent container storage tools from Portworx Inc. and data management tools from Delphix have begun to help ease this burden, as well.
GE Digital put Portworx container storage into production to support its Predix platform in 2017, and GE Ventures later invested in the company.
Now, [developers] make database changes roughly weekly, with 60 developers in 15 teams and 70 application development pipelines.
Onder AltinkurtIT product manager, ING Bank Turkey
"Previously, we had a DevOps process outlined. But if it ended at making a call to GE IT for a VM and storage provisioning, you give up the progress you made in reducing time to market," said Abhishek Shukla, managing director at GE Ventures, based in Menlo Park, Calif. "Our DevOps engineering team also didn't have enough time to call people in IT and do the infrastructure testing -- all that had to go on in parallel with application development."
Portworx allows developers to describe storage requirements such as capacity in code, and then triggers the provisioning at the infrastructure layer through container orchestration tools, such as Mesosphere and Kubernetes. The developer doesn't have to open a ticket, wait for a storage administrator or understand the physical infrastructure. Portworx can arbitrate and facilitate data management between multiple container clusters, or between VMs and containers. As applications change and state is torn down, there is no clutter to clean up afterward, and Portworx can create snapshots and clone databases quickly for realistic test data sets.
Portworx doesn't necessarily offer the same high-octane performance for databases as bare-metal servers, said a Portworx partner, Kris Watson, co-founder of ComputeStacks, which packages Portworx storage into its Docker-based container orchestration software for service-provider clients.
"You may take a minimal performance hit with software abstraction layers, but rapid iteration and reproducible copies of data are much more important these days than bare-metal performance," Watson said.
The addition of software-based orchestration-to-database testing processes can drastically speed up app development, as Choice Hotels International discovered when it rolled out Delphix's test data management software a little more than two years ago.
"Before that, we had never refreshed our test databases. And in the first year with Delphix, we refreshed them four or five times," said Nick Suwyn, IT leader at the company, based in Rockville, Md. "That has cut down data-related errors in code and allowed for faster testing, because we can spin up a test environment in minutes versus taking all weekend."
The company hasn't introduced Delphix to all of its development teams, as it prioritizes a project to rewrite the company's core reservation system on AWS. But most of the company's developers have access to self-service test databases whenever they are needed, and Suwyn's team will link Delphix test databases with the company's Jenkins CI/CD pipelines, so developers can spin up test databases automatically through the Jenkins interface.
Dig Deeper on Systems automation and orchestration
