How Lufthansa is flying its data warehouse to the cloud
Moving from an on-premises data system to the cloud can be a complex operation. Lufthansa is looking to remove some of the complexity with virtualization.
The COVID-19 pandemic hit the global aviation industry particularly hard, forcing many airlines to find new ways to optimize operations.
For Germany-based global aviation firm Deutsche Lufthansa AG, the pandemic was a driver for the company to rethink its data strategy to support a more agile approach than what it had been using, particularly for its data warehouse technologies.
The airline had been using on-premises technologies and is now moving to the cloud. Moving from an on-premises data warehouse to the cloud is not a simple lift and shift exercise. To that end, Lufthansa said on Sept. 20 that it is using the Datometry Hyper-Q data virtualization platform to help move to the Microsoft Azure Synapse cloud service.
"We do have a data technology strategy, and what we wanted to achieve is to increase our resilience and scalability options," explained Mirco Bharpalania, senior director for data, analytics and integration at Lufthansa Group.
How Datometry enables the move to a cloud data warehouse
As with any large organization, the airline maintains multiple database and data warehouse technologies across its operations.
For the cloud migration move, one of the first on-premises systems Lufthansa's shifting is its customer data system running on Teradata.
The move away from its existing on-premises data warehouse systems is part of Lufthansa's larger data strategy. Lufthansa looks to build a unified data platform in the cloud that consolidates its data efforts. The company chose to largely standardize on the Microsoft Azure stack, Bharpalania said.
The Datometry platform -- which the San Francisco-based vendor markets as a specialized technology for Teradata-to-cloud data warehouse migrations -- provides a virtualization layer that abstracts the differences. It lets SQL operations that were running on Lufthansa's existing on-premises system run without problems on its new cloud deployment.
By using Datometry, Lufthansa can reduce its risks in migrating to the cloud; its teams don't have to rewrite every SQL query that they were using on premises, Bharpalania said.
"So the intention is to shift to the cloud and have the virtualization layer of Datometry around it," he said. "The virtualization reduces the risk and increases the speed. Otherwise we would have needed a lot of experts to make the move."
Preparing workloads to move to the cloud data warehouse
As part of the migration effort, Lufthansa first engaged in a thorough evaluation to determine if the Datometry approach would in fact work to satisfy the airline's needs.
We do have a data technology strategy, and what we wanted to achieve is to increase our resilience and scalability options.
Mirco BharpalaniaSenior director of data, analytics and integration, Lufthansa Group
Bharpalania's team did a proof of concept, taking five of its most resource-intensive data pipelines and extract, transform, load workloads and running them in the new environment.
"We had really good results, and that was the first thing we did to safeguard the decision," Bharpalania said. "Now we are in the phase where we're doing optimization."
Lufthansa has been running its existing on-premises systems for decades. The complex process of optimization to get the best possible performance in the cloud is a big and challenging undertaking for Bharpalania's team.
Looking ahead, Bharpalania said Lufthansa's plan is to move more on-premises data warehouse technologies into the cloud, including an existing Oracle deployment. The broader goal with the cloud is to keep data in the same place so the airline can more easily analyze it for analytics and business intelligence as well as to improve operations.
