TD Bank plans AIOps consolidation on Dynatrace SaaS
The dream of zero-touch autoremediation remains alive for the bank as it prepares to go all in on Dynatrace SaaS for observability-driven IT automation.
A multiyear process of adopting AIOps has led TD Bank Group to consolidate its observability tooling on Dynatrace SaaS this year, as it continues to strive for completely hands-off IT automation.
TD Bank, based in Toronto, has used Dynatrace for more than five years already, along with other IT automation tools within a set of hybrid cloud platforms that include a self-managed data center, private cloud and public cloud. But as it shifts more workloads to public cloud and SaaS, the financial services company is preparing to move from a managed instance of the Dynatrace platform on premises to its latest SaaS platform to take advantage of new features such as the Grail data lakehouse back end, automated visualization tools such as Dynatrace Notebooks and prepackaged IT automation apps.
These updates, most of which were rolled out over the last year, prompted a decision to consolidate onto Dynatrace for AIOps by Chris Conklin, head of enterprise observability and operational automation for TD Bank.
"The last year's updates really opened my eyes to the forethought and innovation Dynatrace is focused on -- it's an important intangible when it comes to companies that you partner with," Conklin said. "If they're not innovating, you're going to find yourself several years down the road having to pivot, which is quite costly."
Updates of particular interest to Conklin also included this week's previewed support for AI application monitoring, a data pipeline service and data observability ingestion from third-party tools.
Chris Conklin
"I'm most excited about AI observability," Conklin said from this week's Dynatrace Perform in Las Vegas. "That's a space where, as you can imagine, a bank is incredibly risk-averse. We have to protect our data and protect the customer experience."
OpenPipeline and support for third-party data observability, launched this week, could potentially help identify and block sensitive data from being ingested into such systems, Conklin said.
"That really spoke to me around being able to see the data that's coming through and adjust it," he said. "With some of our other [tools], it requires a lot of effort to find where the exposure is [with private data], then go back and mask [it] and redevelop and redeploy. Just doing that on the fly is really exciting."
The bank now plans to replace specialist tools for networks and other "niche" aspects of infrastructure monitoring with Dynatrace SaaS, although Conklin did not name the vendors. Tool consolidation, particularly in the observability and security markets where Dynatrace competes, has become a common theme amid macroeconomic worries in enterprise IT over the last two years.
'Chipping away' at AIOps
Using existing on-premises Dynatrace tools such as PurePath and OneAgent for automation has already brought TD Bank a few steps into its development of AIOps autoremediation workflows. For example, Conklin began using Dynatrace's Davis AI to create reports for executives on business data five years ago.
"Davis creates one aggregated score that tells them how well technology is delivering critical transactions for our customers," Conklin said. He estimated that the bank now performs 19.5 billion such transactions annually and ingests multiple terabytes of data per day into the Dynatrace system.
Since I started in this role three years ago, that's been the mandate: zero-touch, automated recovery. ... It's a journey, and we're slowly chipping away at it.
Chris ConklinHead of enterprise observability and operational automation, TD Bank Group
Over the last three years, Conklin also wove together Dynatrace AIOps and PagerDuty's incident orchestration software to offer incident response teams a list of automatically suggested actions to resolve an incident. To some extent, Davis AI already provides predictive analytics that can alert IT ops teams at the bank ahead of time about issues such as running out of disk space or application failures that are likely to escalate.
The migration to Dynatrace SaaS, with its newer AIOps features, including business events and insights, will hopefully eliminate that "human in the loop" from the process, Conklin said.
"We currently are doing it more as a prompt versus truly automated, because I would hate to deploy this and immediately take something down on accident," he said. "But since I started in this role three years ago, that's been the mandate: zero-touch, automated recovery. ... It's a journey, and we're slowly chipping away at it."
Dynatrace has won out over other monitoring and observability tools, but Conklin said he doubts any one vendor will serve as the bank's single platform automation engine -- although many, Dynatrace included, claim to offer just that. In TD Bank's case, such vendors include ServiceNow, Ansible and PagerDuty's Rundeck DevOps automation.
"We're putting more and more focus on how we can consolidate things down to remove complexity. ... But it's more [about] finding a best-of-breed tool to do the bits and pieces throughout AIOps," he said. "There's not one tool that can do it all."
Dynatrace execs discussed AutomationEngine and AppEngine updates, including reference guides for platform engineers, during keynotes at Perform.
Dynatrace fleshes out platform automation tools
Enterprises that were customers of Dynatrace before it moved to the SaaS platform, like TD Bank, must undergo a migration process to the new system. Dynatrace officials declined to specify what percentage of the vendor's customer base has made that move as of this week.
Should IT organizations decide to take the plunge with Dynatrace SaaS, however, the vendor will have an expanded set of engineering reference guides and prepackaged applications available to help them set up AIOps workflows for platform engineering.
Company officials who gave keynote presentations at Dynatrace Perform this week touted a set of more than 100 engineering guides for using Dynatrace workflows within its AutomationEngine tool. These include infrastructure management tasks such as automated disk capacity forecasting, application vulnerability prioritization and infrastructure autohealing.
"Some of our early adopter customers are already implementing self-healing to restart processes, autoscale or downscale [infrastructure resources], and they're saving about 20% of their working time on a weekly basis," said Henrik Rexed, senior staff engineer at Dynatrace, during his presentation.
Other presenters discussed new Dynatrace apps built with its customizable AppEngine, including a security threat investigation tool set to ship in April, a business workflow app in preview called Business Flow, and a Kubernetes pod scaling app that shipped this week.
"Our new Dynatrace Kubernetes app ... can quickly identify over- and underprovisioned workloads by detecting CPU and memory slack," said Bonifaz Kaufmann, vice president of automations at Dynatrace. "Soon we will enable you to automate Kubernetes configuration changes in a GitOps fashion to automate pod resizing and other important things in Kubernetes."
Beth Pariseau, senior news writer at TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Dig Deeper on Systems automation and orchestration
Chris Conklin
