DBT Labs launches AI copilot to boost developer efficiency

AI for efficiency

Generative AI has the potential to transform business, making employees better informed with data accessed through natural language interfaces and more efficient with automation. As a result, enterprise interest in AI development has surged since OpenAI's November 2022 launch of ChatGPT marked a significant improvement in generative AI capabilities.

Generative AI development, however, is complex.

It requires engineers to combine proprietary data with not only generative AI models so the tools built understand the unique characteristics of an individual enterprise, but also with governance so data remains secure and used according to both organizational and regulatory guidelines.

Substantial amounts of documentation, semantic modeling and SQL formatting are required throughout development. In addition, tests need to be run repeatedly to ensure accuracy, consistency and compliance.

With enterprises wanting rapid returns on their investments in AI development, tools that can reduce the time developers and engineers spend on repetitive tasks are beneficial.

The opportunity to relieve developers and engineers of repetitive tasks provided part of the impetus for DBT's development of its copilot, according to Mark Porter, the vendor's chief technology officer. In addition, some customers requested such capabilities.

"Our understanding of DBT's models, lineage and testing frameworks gave us a unique advantage to build something contextual," Porter said. "With DBT Copilot, we're leveraging this insider knowledge to help data teams work smarter on problems that actually matter."

Meanwhile, given that DBT's assistant uses AI to aid engineering workflows, it is a significant addition, according to Mike Leone, an analyst at Enterprise Strategy Group, now part of Omdia.

"This new copilot can provide a great opportunity to streamline processes, helping save time and reducing the risk of errors," he said.

Because it integrates with metadata and schema already in use, it won't hurt the precision and accuracy of data pipelines, Leone continued.

"It offers customers a more efficient way to scale their data operations while maintaining data integrity, which is absolutely essential for high-quality analytics and AI-driven insights," he said.

Features of DBT Copilot include the following:

• Standardized automatically generated YAML-based documentation with one click to relieve developers and engineers from having to write model descriptions and column definitions.
• AI-suggested validated testing based on the context of DBT models, so users don't need to craft each test manually.
• SQL formatting and query optimization capabilities that enable users to use natural language that DBT Copilot translates to code according to a built-in style guide to ensure that it's correctly formatted.
• Automatically generated recommendations for semantic layer and metric definitions based on existing data models.
• An integration with Azure OpenAI Service.
• A style guide to standardize SQL formatting to ensure consistency across data and AI projects.

All are generally available with the launch of DBT Copilot other than the style guide, which is in beta testing.

Context-aware documentation and testing will be particularly beneficial for DBT users, according to Leone.

"They ensure data consistency, improve collaboration and proactively catch issues, all while enhancing data quality and pipeline reliability and freeing up teams to focus on strategic work," he said.

Meanwhile, given all the features DBT Copilot includes, it is "impressive" with the recommendations for semantic layer and metric definitions perhaps the highlight, according to Menninger.

"Much of the value of analyzing data comes from the metrics that are derived from the data that is collected," he said. "And these are important elements of the semantic model of an enterprise. If you can't describe and capture the metrics, you really don't have a complete definition of the semantic model."

Plans

With DBT Copilot generally available in DBT Cloud, the vendor's roadmap includes adding more AI tools to improve analytics workflows and the developer experience, according to Porter.

"We plan to embed context-aware intelligence throughout the entire analytics development lifecycle," he said.

DBT's focus on enabling increased efficiency through AI is wise, according to Leone.

"I think DBT should continue to focus on expanding AI-driven optimization, and that includes things like offering real-time recommendations to improve pipelines," he said.

In addition, making it easier for users to customize workflows and integrating more machine learning capabilities could be other ways for DBT to evolve, Leone continued.

Menninger, meanwhile, suggested that DBT needs to make its AI-powered assistant more broadly available. While generally available in the vendor's development environment, it is not yet so in its low-code Visual Editor.

"Making these features generally available in the Visual Editor is necessary to enable more of the business user audience," he said.

Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.