Comcast subsidiary survives March Madness with AIOps
AIOps alert reduction brought order to a chaotic environment at TV advertising specialist FreeWheel just in time for a big test of its resiliency during a wild NCAA tournament.
The annual NCAA basketball tournament is the ultimate unpredictable workload for a TV ad-serving company, but alert reduction via an AIOps tool came through for FreeWheel in the clutch.
The opening night of the NCAA men's basketball tournament March 14 brought in an average of 8.4 million viewers across multiple TV networks, up 2% from last year's tournament and the most since 2015, according to Sports Media Watch. Three nights later, when the relatively unknown Fairleigh Dickinson University became only the second No. 16 seed to defeat a top-seeded team in tournament history when it beat Purdue, prime-time March Madness coverage on CBS averaged 3.35 million total viewers, on par with its performance on opening night, according to TVLine.
Amid all this, Comcast subsidiary FreeWheel had a crucial moment to seize, according to Michael Lorenzo, senior director of operations at the company.
"Live events in the ad-serving world present a unique challenge, because you don't know how many people are going to watch the basketball game," he said. "And if you miss that first opportunity, it's gone forever -- you can never recover that opportunity."
The night of the Fairleigh Dickinson upset was a particularly momentous one for ad serving, Lorenzo said in a recent interview.
All of a sudden, they're winning, users log on, and the opportunity for [ad] impressions just grew by tenfold.
Michael LorenzoSenior director of operations, FreeWheel
"All of a sudden, they're winning, users log on, and the opportunity for [ad] impressions just grew by tenfold," he said.
It's no time for IT systems to fail. But had this occurred during last year's March Madness, the likelihood of an extended outage would have been much higher, according to Lorenzo. At that time, FreeWheel, which had been assembled out of a series of acquisitions by parent company Comcast over the last six years, had up to a dozen IT monitoring tools sending support staff and IT operations engineers as many as 15,000 notifications per day via email.
"'Let's get this out of email' was a challenge that had been put on me for the past year," Lorenzo said. "You have all these different engineering teams, and they're used to doing things a certain way, so changing their workflows is very difficult."
From a tangled web to nothing but net
Starting in June, FreeWheel tested multiple AIOps tools, which Lorenzo did not name, and settled on BigPanda, which it began using in November to ingest data, correlate events and reduce alerts from its various monitoring tools. Within the first two months, it had reduced those 15,000 alerts by 71% and reduced its mean time to recover from 25 hours per incident to 5.5 hours.
"A lot of the other vendors that we looked at wanted to solve all of our AIOps challenges in one step," Lorenzo said. "And we knew we needed to crawl, walk and then run -- we wanted to get noise reduction first."
BigPanda's tech support helped integrate FreeWheel's on-premises Atlassian Jira Server environment during its initial deployment, before the company moved completely to Jira Cloud by the end of 2022. This was crucial to being able to do the initial rollout of the AIOps tool, and BigPanda's support engineers were accommodating and helpful with this work, Lorenzo said.
Using BigPanda has also helped FreeWheel start to optimize its event and alert routing via Jira tickets among three tiers of IT teams: a tier 1 global support organization consisting of 120 people, 15 of whom operate the company's network operations center; a tier 2 team of 10 to 15 IT ops engineers, including an on-call site reliability engineer; and a tier 3 pool of the company's more than 700 software developers.
"In the past, that was mainly done with conversations in Slack, which worked -- the teams were able to look up and see what they did in the past," Lorenzo said. "But it didn't give us any real reporting on how much work we were doing or how much distraction we're [giving] a developer who's now doing support."
Auto-remediation and BizDevOps hoop dreams
In the future, FreeWheel plans to expand into BigPanda's unified analytics products to get better insights into business performance metrics along with IT systems. FreeWheel is also still working on consolidating monitoring tools onto a Prometheus time-series monitoring system and integrating BigPanda into its AWS and Kubernetes environments. It aspires to automated remediation with AIOps as well, but this remains a work in progress, as does aligning the company's data asset tagging with BigPanda to improve correlations on the back end.
Michael Lorenzo
FreeWheel could instead customize the BigPanda system to ingest data from its systems as is, rather than changing its own data tagging schemes, but it's important to have BigPanda's algorithms operating on data as efficiently as possible, Lorenzo said.
"Our focus was, let's not take a tool and customize it, and add all of these intricacies to it," he said. "Let's focus on what's wrong in our system first, fix that."
In the meantime, March Madness 2023 has gone off without a hitch, Lorenzo said.
"We've been able, as an organization, to run very smoothly for our second March Madness -- this has helped take some of the guesswork out of it," he said. "Just the amount of effort I would say that we're putting into the eyes-on-glass type of monitoring ... [is lower] because we're able to respond to the incidents coming into BigPanda rather than looking in multiple places."
Beth Pariseau, senior news writer at TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Michael Lorenzo
