Monte Carlo introduces Circuit Breakers for data pipelines
In an effort to help improve data workflow reliability, Monte Carlo is rolling out a new feature that can help organizations stop inaccurate data flows from affecting operations.
Data observability vendor Monte Carlo rolled out a new Circuit Breakers feature this week to help prevent broken data pipelines from negatively affecting applications that rely on the data.
Monte Carlo's Data Observability platform helps organizations better understand and monitor data pipelines. In 2021, the San Francisco-based company rolled out a series of initiatives including Monte Carlo Incident IQ to help data professionals determine why a given data flow isn't working.
The new Circuit Breakers feature allows data teams to stop a running data pipeline if the service detects that the data is in some way inaccurate or faulty.
Data pipelines increasingly power critical operations
Organizations use data pipelines for a specific purpose. That purpose could be to inform a business report, operational dashboard or data analyst, or to train a machine learning model, explained Lior Gavish, co-founder and CTO of Monte Carlo.
Increasingly, data pipelines power business-critical operations where, according to Gavish, if the data is wrong, there could be measurable negative effects on the organization. The Circuit Breakers feature is configured with policies that organizations can use to define data pipeline quality.
Given the importance of data pipelines for certain operational uses, organizations need the ability to stop a broken pipeline before it propagates bad data into the business process. That's where Circuit Breakers fit in, Gavish said.
"In our observability platform, we allowed our customers to monitor data and get alerts on things, which is great," Gavish said. "But we didn't actually change the data pipelines or the way they run if something bad happened and the wrong data was present."
Monto Carlo is adding Circuit Breakers to its data pipeline observability platform with the ability for organizations to set policies for when a pipeline should be stopped.
Preventing data pipeline issues
One Monte Carlo platform user is South Carolina-based Red Ventures and its Red Digital division, which provides marketing and data services to clients. Brandon Beidel, director of product management, data platform, at Red Ventures, explained that his team spent an inordinate amount of time chasing bugs in data pipelines. That challenge led Beidel to discover Monte Carlo and its Data Observability platform for data pipelines.
Circuit Breakers are going to allow us to essentially inject integration tests into our full data pipelines so we can check our assumptions before we end up putting issues into reporting.
Brandon BeidelDirector of product management, data platform, Red Ventures
"The core of the value we were trying to get at was how do we increase the trust in our data so we can look at forward-looking tasks and not always be looking to question the information that's being provided to us in our data systems," Beidel said.
Beidel said his team has been using the Monte Carlo platform for the last six months and recently tried out the Circuit Breakers functionality as part of an early customer preview.
According to Beidel, the Circuit Breakers feature could help his team to detect issues earlier than they would otherwise. The new Monte Carlo features allow his team to inject critical assumption tests about the state of data and data pipelines into the data workflow process. With those tests in place, it's possible for Beidel's team to stop a data pipeline process early on if there are data quality issues that will affect operations.
"Circuit Breakers are going to allow us to essentially inject integration tests into our full data pipelines so we can check our assumptions before we end up putting issues into reporting," Beidel said.
