Fivetran updates data integration platform with dbt Core
With automated data transformations and new data lineage visualization graphs, Fivetran is looking to make building data pipelines easier for data users.
Fivetran today released its updated platform with enhanced capabilities for integrating data transformations with the open source dbt Core technology.
Fivetran, based in Oakland, Calif., is a data integration platform vendor that provides users with connectors to link data sources to other systems for business intelligence and data analytics.
As part of the data integration process, organizations often need to transform data so that it fits into a structure and format the target destination can use. Open source dbt (data build tool) technology, an increasingly popular approach for data transformations, comes into play as part of that process.
Fivetran released a dbt Core integration in September 2020 and has now updated the technology with enhanced capabilities. Among the new features are integrated data lineage graphs as well as automated scheduling for dbt Core transformation workflows.
Data teams need faster, easier ways to consolidate and reconcile multisourced data sets. Fivetran and dbt Core address these requirements by helping data engineers schedule the transformation of data as it arrives from various sources.
Kevin PetrieAnalyst, Eckerson Group
As enterprises grow, they add data sources, users and applications. That increased scale creates complexity and complicates data pipelines, said Kevin Petrie, an analyst at Eckerson Group.
"Data teams need faster, easier ways to consolidate and reconcile multisourced data sets," Petrie said. "Together, Fivetran and dbt Core address these requirements by helping data engineers schedule the transformation of data as it arrives from various sources."
Scheduling data integration with dbt Core transformations
Alexander Lovell, head of product for connectors and transformations at Fivetran, said that the initial iteration of the dbt Core integration didn't have an optimal approach to scheduling.
Fivetran's previous dbt Core integration ran on a time-based schedule in which users needed to configure the day and time a dbt Core data transformation would run. Users had to guess when data was updated in order to decide the right time to trigger the dbt Core transformation, Lovell said.
Fivetran is now providing an automated integration for dbt Core. As soon as the source data updates, it can trigger a dbt Core data transformation as part of the data pipeline.
Fivetran now enables an automated approach for data transformation with dbt Core that executes when new data is present.
"With this new integration, instead of having to start a transformation on a time basis, where users are guessing at when the data will be available, it's automated within the system," Lovell said. "So now when data is ready, we kick off a dbt Core transformation based on what's actually happened in the system rather than just a time-based approach."
Data lineage provides visualization for data integration
Another key part of the dbt Core update in Fivetran is a data lineage graph capability.
Data lineage enables users to track where data came from and how it was used in a data pipeline. Fivetran added data lineage visualizations with graphs to make it easier for users to understand and monitor the data integration process as data moves from the source to the destination through the dbt Core data transformation model.
Lovell noted that data lineage graphs provide a visual way for data analysts to understand how data pipelines are working.
The vendor's overall goal is to make data analysts' jobs easier so they can focus on the work that drives business value, such as data analysis, rather than spending more time on building data pipelines, Lovell said.
_mobile.jpg)
