IT orgs expand business observability beyond their own walls

FreedomPay sketches out business observability outreach

The impetus for this expansion came in October, when Dynatrace made automatic correlations between log data and distributed traces generally available on its Grail data lakehouse back end, introduced in 2022. Another metric called Business Events, also known as bizevents, was also introduced in 2022. Business Events draws on the Dynatrace OneAgent API to gather data from third-party ERP, CRM and other business information systems, along with Dynatrace real user monitoring (RUM) tools. The result is the ability to make connections between the performance of IT systems and applications and business functions such as purchasing, shipping and product delivery, as well as whether these processes were successful from a consumer's point of view.

October's log and trace integration on Grail opened the door for new ways to analyze Business Events, according to Tomlinson.

"Bizevents records represent 100% accurate captured data -- not aggregated," Tomlinson said in an email interview with Informa TechTarget following Dynatrace Perform. "Then, for response times, you can join bizevents records to their associated [distributed tracing] spans to retrieve an aggregated calculation of response times."

This ability to provide specific, unabridged, fine-grained data analysis on individual business and application processes has Tomlinson working to push Dynatrace OneAgent into customer and partner environments, along with alerting and Davis AI anomaly detection.

For example, if there are 1,600 business transactions recorded in five minutes, the new Dynatrace Grail log and trace integration can join them with application performance data and calculate an average response time to consumers during that five-minute period, Tomlinson said. That data in turn can be shown to FreedomPay's partners, such as credit card issuers, and customers, such as Starbucks, on the new shared dashboard. The dashboard will directly alert those third parties about whether transaction response times are within a desired range.

Ultimately, Tomlinson said he envisions this shared web of business observability driving new business process automation systems that can respond quickly and automatically to issues.

"You could configure this whole thing using logs instead of bizevents," he said. "[But] the mechanisms for capturing a bizevent are instrumented on the request processing pipeline prior to any customized application code that may crash or have bugs that compromise its ability to even write its own data. … I believe that observability data captured from the surrounding infrastructure supporting an application can be more truthfully accurate than the application's own system of record data."

FreedomPay hammers out RBAC for shared dashboard

For now, work is still in progress to put the finishing touches on a complex set of role-based access controls between FreedomPay's systems and Dynatrace, according to Christopher Kronenthal, president of FreedomPay, in an interview with Informa TechTarget at Dynatrace Perform.

Christopher Kronenthal

"I already have a way that I manage customers in a [multi-tenant] fashion," Kronenthal said. "I need hooks into a Dynatrace that allow me to shard out that data [access] but in a way that … feeds up into both tenantized and non-tenantized assets. You can do that [now], but it's more cumbersome than it needs to be."

Tomlinson said his goal for 2025 is to refine this integration into a polished product with these security details ironed out. Dynatrace's data gathering features also offer a security advantage, he said.

"Bizevents [are] all just metric metadata around the core payload -- for us, [that's] payment data that is sacrosanct, encrypted, untouched -- where Dynatrace's data privacy abilities are perfect for highly secured end-to-end systems," he said.

Dynatrace officials did not respond to a request for comment on this topic as of press time.

Mitchells & Butlers pulls partners into business observability

Mitchells & Butlers, a bar and restaurant company in the U.K., was among the earliest adopters of Business Events in 2022. At the time, IT observability tools could alert operations teams about whether a technical service was up or down, said Mark Forrester, digital readiness manager at the company, during a breakout session at Perform. But Business Events added information about subtler service degradations and intermittent failures that might still affect consumer experiences, he said.

For example, Forrester said, if a restaurant used a website to book tables, but it's slow to respond or doesn't work, it was difficult to tell how many consumers might use the fallback method of calling the restaurant instead. Or an online table booking might work, but a digital menu might fail to load. Mitchells & Butlers expanded its business observability data pool using Business Events to get a clearer sense of how often these kinds of failures happen and how consumers react, pinpoint the causes of those failures and remediate them.

"We've got 1,600 outlets that have the ability in real time to set items 'out of stock,'" Forrester said. "We've got to take that real-time change in a physical, third-party SaaS [point of sale service] all the way through the digital layer, and then integrate that with other parties, including third-party delivery integrators … like Uber Eats."

This meant those third parties, which also include table booking, table management and digital payment providers, had to agree to exchange Business Events data with Mitchells & Butlers. They were convinced to collaborate by the prospect of sharing the burden of operational resilience, Forrester said.

"We have a new third-party [partner] that does the integration [with delivery services] and I said, 'Oh, this is great, but what's your support team like?' Because every order that fails, you know why it failed, I don't, so every time a [restaurant] says, 'My order hasn't gone into the [system] automatically,' we're going to be logging an incident to you," he said. "[The partner] went a funny shade of white as he realized the massive workload that was coming his way."

Instead, the SaaS provider partner now sends Mitchells & Butlers Business Events data via the Dynatrace OneAgent API, Forrester said.

"They'll tell us what's happening with our orders, and if it fails in their world, they've got their hands raised, saying, 'We made that fail.'" A steady stream of third-party Business Events and anomaly detection in Dynatrace's Davis AI also helps keep track of how Mitchells & Butlers APIs are performing for external users, Forrester said.

What's next for Business Events

Mitchells & Butlers doesn't use the new log data integration on Dynatrace Grail yet, but RUM on Grail alerts support teams to degradation in the consumer experience at restaurant and bar locations. From there, that team can dig into correlated IT and app observability data and Business Events on Grail to determine how widespread the failure is and its root cause.

The company will start using Dynatrace's OpenPipeline to collect data, including from logs, but Forrester said he plans to extract data from logs and store it as Business Events long-term to save on data retention costs.

"If there's information in those logs, strip it out into useful business events," he said. "Most of it will be information that no one will ever need anyway. Strip out what you need, drop the rest and you can save some money."

This approach will be necessary as Mitchells & Butlers sends Business Events from its Apigee API gateway into Dynatrace this year, which Forrester estimated will account for 1 billion events annually. Forrester said he'd also like a direct pipeline for data from Dynatrace Business Events into Microsoft's PowerBI business intelligence reporting tools. Dynatrace officials said the company has built this integration and is awaiting certification by Microsoft.

In the next few months, Dynatrace will add a new toggle in its Business Flow app that makes it easier to configure Davis alerts on Business Events data, according to Klaus Enzenhofer, principal product manager at Dynatrace, who co-presented with Forrester at Perform. The Dynatrace problem card for troubleshooting will correlate Davis alerts with data in Business Flow to help operations teams prioritize responses.

In the past, the business impact analysis panel in the Problem card showed the number of affected users and the number of associated service calls, but not which business processes were affected.

When the problem card shows an issue affecting "one front-end application, three services [and] five infrastructure pieces, [users asked,] 'Are these serving our company's about page, or is [a] critical service impacted here?'" Enzenhofer said. "We are now actively putting in there if there is a business process impacted [and] a quantification of what it means from a revenue perspective."

Beth Pariseau, senior news writer for Informa TechTarget, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.