Alation unveils enhanced partnerships with Databricks, DBT

New integrations

Alation had a preexisting connector to Databricks that enabled joint customers to organize and access their data stored in Databricks lakehouses for use in analytics and data science projects.

The enhanced partnership between the vendors adds connectivity to Databricks' Unity Catalog, an in-lakehouse data catalog that enables organizations to put in place data governance and data lineage measures. The Unity Catalog was first released in preview in 2021 and made generally available in June 2022.

It's the addition of data and analytics assets stored in lakehouse to Alation's own data catalog -- and ability to access those data and analytics assets with Alation -- that makes the enhanced integration significant, according to Stewart Bond, analyst at IDC.

The integration ... [provides] better visibility into the Databricks environment for business data analysts, and it provides a more comprehensive view of all data assets for incorporation into data lineage and governance processes.

"The integration ... [provides] better visibility into the Databricks environment for business data analysts, and it provides a more comprehensive view of all data assets for incorporation into data lineage and governance processes," he said.

He added that Databricks' Unity Catalog is built for a technically savvy target audience of data experts, while typical data catalogs such as those built by Alation and Collibra are often aimed at self-service users who need to find and work with trusted data without having to write code.

The new connector addresses the needs of the data engineers and data scientists who are sometimes not served by Alation's existing data catalog.

"The Unity Catalog in Databricks is very technical and serves a different persona compared to the typical persona using the Alation catalog," Bond said.

Alation also had a preexisting relationship with DBT Labs that goes back to October 2022 when the two formed their initial partnership. Now, Alation is adding a new connector that provides support for DBT Core and DBT Cloud in Alation.

DBT Core is an open source tool that enables data engineers to transform their data. DBT Cloud is the hosted version of DBT's platform, a managed service that provides a web-based UI for data transformation.

The connector will enable joint users to extract descriptions for curated data and data lineage for data warehouses such as Snowflake and PostgreSQL out of DBT's environment. They can then move the descriptions and lineage into Alation to more easily view and understand transformations in their underlying data.

"The integration with DBT is significant in that it provides data engineers with better descriptions of models and columns and lineage of data," Bond said. "It also provides a facility that DBT users can leverage to manage and control DBT asset groups, providing the opportunity to include DBT assets into broader data governance processes."

Taken together, the result of Alation's new connectors to Databricks and DBT Labs is a better overall understanding of an organization's data ecosystem, he continued.

"The new capabilities provide Databricks, DBT and Alation users with better context, deeper connectivity, lineage and visibility across the modern data environment, providing a deeper level of intelligence to all," Bond said.

Diby Malakar, Alation's vice president of product management, said the new connectors are aimed at expanding Alation's connectivity and giving users greater visibility into their data. He added, however, that both Databricks and DBT Labs were targets for new connectors because of the rising importance each is playing in the development of the modern data stack.

"One of the common patterns we see is people wanting to get data from legacy systems into a modern data system, and then use DBT to transform the data," Malakar said.

In particular, DBT is gaining momentum, he continued.

Snowflake is perhaps Databricks' main competitor and is as popular as Databricks. As a result, Alation has a close partnership with Snowflake.

DBT, meanwhile, is rising in popularity among Alation users, according to Malakar.

"DBT is playing a very dominant role in transforming data in the [extract, load and transform] paradigm," he said. "More and more, we're seeing the rise of modern data stack vendors, and we felt Databricks and DBT were two dominant players in that landscape that we definitely want to support in the context of supporting self-service."

The road ahead

Alation rolls out updates once per month.

Moving forward, Malakar said the vendor's roadmap features more integrations with DBT and that Alation plans to integrate even more tightly with the Databricks Unity Catalog's data lineage features. In particular, as DBT Cloud gains popularity, Alation expects to provide more synergy between its own data catalog and DBT's managed service.

"As we see [customers] doing more in the cloud as part of their cloud transformation, we feel like the integration we're building with DBT in the cloud will serve as an accelerator for them," Malakar said.

Bond, meanwhile, noted that Alation's Open Connector Framework enables the vendor to quickly build connectors that give its customers access to new technologies as they emerge.

The new connectors to Databricks and DBT are both examples, with the new Databricks connector adding access to the lakehouse vendor's Unity Catalog, and the DBT connector adding access to an emerging vendor's suite of capabilities.

Moving forward, Bond expects that Alation will continue adding connectors that enhance what its customers can do with its platform.

"As customer demand shows up for new technologies that emerge in the modern data environment, it should not take long for Alation or third parties to add new connectors," Bond said.

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.