American Airlines lowers data management costs with Intel
As the airline giant moves more of its data workloads to the cloud, tools from Intel's Granulate are making platforms such as Microsoft's Azure Data Lake more efficient.
With data critical to the airline industry, American Airlines is deploying tools from Intel to help keep its data management costs under control.
Nearly everything the airline giant and peers such as United Airlines and Delta Airlines does relies on data, from staffing planes to scheduling flights and adjusting in real time as weather, mechanical problems and other factors alter plans.
Meanwhile, American Airlines and its competitors collect massive amounts of data with thousands of flights carrying hundreds of thousands of passengers taking place every day.
Storing all that data is expensive. So is ingesting, integrating, observing, querying, modeling and analyzing all that data. With cloud-based data platforms charging based on use, data management costs can accumulate quickly and often exceed expectations.
Cost control and optimization are priorities, according to Rasika Vaidya, director of cloud and engineering platforms at American Airlines, who spoke recently in an on-demand session during Microsoft Ignite, a user conference hosted by the tech giant.
To keep those costs as low as possible, American Airlines is using workload optimization tools from Granulate, which is now part of Intel.
"We have looked at our cloud [costs] over the past few years to find efficiencies," Vaidya said. "Intel has been with us, focusing on our business challenges and trying to figure out how we can solve them."
Like many enterprises, American Airlines stored its data on premises before the advent of the cloud. Now as it modernizes and migrates much of its data to the cloud, the airline giant uses Microsoft Azure, including Azure Data Lake, for some of its data management.
The data lake enables data management and analytics that result in data-informed decision. But there's a price to pay for the agility enabled by the data lake and other cloud-based data management tools.
Rasika Vaidya, director of cloud and engineering platforms at American Airlines, speaks during an on-demand session at Microsoft Ignite, a user conference hosted by the tech giant.
The problem
The 2022 holiday season provided a striking example of what can happen when an airline doesn't invest in its technology infrastructure, including data management and analytics.
A winter storm hit parts of the United State the day after Christmas, causing widespread flight delays and cancellations. Quickly, however, most airlines reacted and were able to return to normal within a couple of days. The exception was Southwest.
By Dec. 27, Southwest accounted for 84% of the nation's cancellations. By Dec. 29, the airline accounted for 99% of the nation's cancellations. Three days after the storm, all other airlines were essentially operating normally; Southwest was still unable to get back on track.
Much of the blame was attributed to Southwest's lack of a modern technology infrastructure, including data management. The airline acknowledged at the time that it "fell short."
American Airlines has invested in modern data management and analytics tools. But these come at a cost, becoming more expensive the more they are used.
Usage for Azure Data Lake's analytics capabilities is divided into unit hours. True pay-as-you-go customers are charged $2 per unit hour. However, Microsoft offers packages that reduce the cost. Those packages start at 100 unit hours for $100 and extend up to 100,000 unit hours for $52,000.
The cost savings are substantial for enterprises that know they're going to spend significant time using the platform for their data management and analytics needs. But for massive corporations like American Airlines, 100,000 unit hours is only a beginning.
Given that not all data use is pre-planned and, therefore, part of a finite budget, cloud computing costs can sometimes go well above expectations.
In addition, beyond paying for the time spent working with data, customers must pay for data storage in Azure Data Lake, which is charged based on the capacity customers use.
Therefore, a bit less than two years ago, as American Airlines moved even more of its data from on premises to the cloud, the airline giant made a concerted effort to keep cloud-based data management and analytics costs under control.
"For the past 18 months or so, we have been focusing on laying the foundation for a cloud center of excellence," Vaidya said. "The key priorities for this area have been around cloud governance and optimizations and efficiencies."
American Airlines' initial target for cost optimization was Azure Data Lake, according to Vijay Premkumar, senior manager of public and private cloud at the airline.
He noted that the airline was expanding its use of Azure Data Lake, which was resulting in rising costs.
"Our primary objective was to adopt the cloud in a very efficient way," he said. "We started with a robust solution in Azure Cloud, a robust data lake solution. It grew exponentially. It created a good ecosystem for our application teams to work with data. But it came with challenges."
Specifically, the costs associated with using Azure Data Lake were rising, Premkumar continued.
The solution
American Airlines and Intel have been technology partners for more than 20 years, according to Melvin Greer, Intel's chief data scientist of the Americas.
Intel is a provider of central processing units and semiconductors, essentially providing the base on which many technology infrastructures are built. American Airlines is heavily reliant on mainframes, according to Vaidya.
We have looked at our cloud [costs] over the past few years to find efficiencies. Intel has been with us, focusing on our business challenges and trying to figure out how we can solve them.
Rasika VaidyaDirector of cloud and engineering platforms, American Airlines
Therefore, with American Airlines' use of Azure Data Lake on the rise and the cost of using the platform increasing along with rising use, the airline approached Intel to see if the software giant could help American Airlines reduce computing costs. If couldn't provide the help itself, perhaps Intel could at least provide guidance.
At the time, Intel was partners with Granulate, a specialist in optimizing the performance of data centers, particularly those in the cloud. Intel connected American Airlines with Granulate, according to Premkumar.
"When we shared our challenges, [Intel] said it didn't have a solution at the time but through its partner program [is associated] with a company called Granulate," he said. "We could then see if they fit our needs."
Granulate's tools don't address usage that can result in high data management costs. That must be addressed with access controls and other data governance measures.
Instead, Granulate's platform automatically observes and learns usage patterns and data flows to identify where resources are stressed and bottlenecks occur. These lead to slow run times and opportunities to prioritize workflows.
American Airlines and Granulate began working together after being introduced by Intel. But beforehand, American Airlines looked at two other vendors whose platforms aim to help customers control cloud computing costs by optimizing data workloads.
Premkumar did not name the two vendors American Airlines considered in addition to Granulate but said the results were better with Granulate.
Other continuous optimization software vendors include Akamas, Anodot, Densify and Site24x7.
"Like any enterprise, when bringing in a new solution, we were able to run benchmarks and compare Granulate to other competitors," Premkumar said. "We saw which would provide the best outcome for American Airlines. Granulate stood out miles ahead of the other two."
Since implementing Granulate, American Airlines has seen a noticeable difference in its performance using Azure Data Lake.
The company did not specify what it was spending on Azure Data Lake before partnering with Granulate nor compare that to what it is spending now. But American Airlines did say it has improved Azure Data Lake's performance by 20% while simultaneously optimizing usage on the airline's part.
"We took our challenge to them, and … they got it," Premkumar said.
Beyond reducing data management costs, the American Airlines data team was relieved of pressure to lower spending by eliminating ad-hoc use that could lead to discoveries, according to Vaidya.
"No one likes it when they have to sit before senior leaders and explain why the budget is rising the way it was," she said. "In this case, the solution was a win-win. [The senior leaders] were as excited as we were."
Intel acquired Granulate in March 2022 for an undisclosed sum, later reported to be $650 million. The 2018 startup had raised $45 million in funding and was valued at about $150 million before its acquisition.
Now, with cloud cost control becoming a concern for many organizations, Granulate is a significant capability within Intel's broad swath of offerings, according to Greer.
Future plans
American Airlines' success using Granulate to reduce spending on Azure Data Lake served as proof of concept for the airline giant.
Moving forward, American Airlines plans to extend its use of Granulate to improve the performance and lower the costs of using other data management tools within its vast data operations, according to Premkumar.
"What we achieved in the cloud data lake solution was great," he said. "But we have a long road to go."
That includes workloads in other tools within Azure Cloud, Premkumar continued. In addition, American Airlines has workloads in numerous data centers beyond Azure.
But further use of Granulate isn't the only way American Airlines plans to lower data management costs, according to Vaidya. The airline giant is also looking at how AI can play a role in making its data operations more efficient.
AI can automate workloads that previously had to be done manually. In addition, the natural language processing capabilities within generative AI tools that reduce the need to write code or, in some cases, eliminate it entirely.
That can enable more people within organizations to do self-service data preparation and analysis, which frees data experts from having to do all of their organization's analysis. It also improves data experts' efficiency by eliminating the time-consuming task of writing code for every action.
"I'm excited about the role AI can play at American Airlines," Vaidya said.
Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.
