Databricks partners with Anthropic to aid GenAI development

The partnership

With generative AI applications capable of making workers better informed and more efficient, many enterprises have increased investments in developing such tools since OpenAI's November 2022 launch of ChatGPT.

However, for large language models (LLMs) such as Claude to understand a business and assist workers, developers need to fine-tune the models with proprietary data.

Without native integration, the fine-tuning process includes exporting data out of a data management platform's secure confines to the AI model or importing the model into the data management platform through an API. Both carry security risks, and data egress costs can add up.

Native availability eliminates the need to move data or models, enabling developers to more easily, securely and inexpensively combine proprietary data with generative AI models. As a result, the partnership with Anthropic is a significant one for Databricks users, according to Constellation Research analyst Andy Thurai.

"While Databricks has become the de facto standard for data lakes, which can be used to train and inference the models, the users have to depend on their own work to integrate with LLMs," he said. "This partnership provides a means to tightly integrate them together."

One concern people have is that they don't want to have to use different services with their own governance, so the fact that [data and AI] are governed through the Unity Catalog is a huge benefit.

In addition to unified governance, another Databricks-Anthropic partnership benefit is simplified development of autonomous generative AI agents using Claude's advanced reasoning capabilities.

Different LLMs have different strengths. Claude is known as one of the better models for generating the code required when developing applications, Mohan said.

The partnership, therefore, is particularly beneficial to developers generating code to build agents and other AI tools, according to Thurai.

"Building complex agents with multistep workflows and a reasoning engine in the back is not an easy task," he said. "This combo will help them move quickly."

Mohan likewise called out simplified agentic AI development as a major benefit of the partnership.

"This is a well-planned, well-thought-out integration, from the foundation up to the user access level," he said. "Anthropic has made an excellent name for itself and has a strong reputation as one of the better models for code development."

Regarding the impetus for the partnership with Anthropic, the rise of agentic AI and customer desire to build agents were motivating factors, according to Hanlin Tang, CTO of neural networks at Databricks.

"[Enterprises] are looking to build domain-specific AI agents with their proprietary data. Anthropic's Claude models excel at advanced reasoning use cases with complex multistep workflows," he said.

From a competitive standpoint, although beneficial to Databricks users, the partnership with Anthropic does little to boost Databricks in relation to rival Snowflake and tech giants AWS, Google Cloud and Microsoft, according to Mohan.

All are racing to provide users with the most advanced tools for generative AI development and model management. Snowflake was perhaps the slowest to react following ChatGPT's launch. But at this point, all are providing similar development and management environments.

"Development environments are pretty standard now ... but it's still early days," Mohan said. "I give Databricks very high marks for innovation. But the innovation is coming from the cutthroat competition in both the data ecosystem and the AI ecosystem -- that's what is pushing these companies to consistently innovate."

Thurai, meanwhile, posited that because Databricks was faster than Snowflake following ChatGPT's launch, its development suite remains more advanced. Additionally, enterprises rarely use both data management and AI development tools from hyperscale cloud vendors, giving Databricks an edge over them as well, he continued.

"Databricks has established itself as a leader in the GenAI model development and deployment space," Thurai said. "Snowflake is a distant second."

Next steps

Beyond providing customers with more choices during AI development, simplifying agentic AI development is a focal point for Databricks over the next few months, with its annual user conference scheduled for June, according to Tang.

Thurai noted that Databricks has been establishing partnerships that expand its ecosystem and is quickly adding new features that enable building and deploying AI applications through acquisitions and internal product development. The vendor's aim to provide new features enabling agentic AI development is expected, he said.

Mohan, meanwhile, suggested that Databricks do more to unify its platform to make it easier to navigate. The vendor provides the necessary tools for data management and AI development, but the tools require engineering expertise to piece together.

"I'm looking to see them further push their unified approach to data and AI," he said. "They need one single stack for structured and unstructured data, for batch and streaming data workloads, for traditional analytics and generative AI applications -- one easy-to-use stack that scales."

Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.