Databricks Apps a toolkit that simplifies AI development
Features including native integrations with development platforms and automatic provisioning of serverless compute enable customers to more easily develop and deploy applications.
Databricks on Tuesday unveiled Databricks Apps, a suite of features that aims to make it easier for users to build customized data and AI applications.
Databricks already provides Mosaic AI, an environment that enables customers to integrate systems such as large language models (LLMs) with their enterprise's proprietary data. Missing, however, were the capabilities to develop the interactive applications such as generative AI chatbots that are powered by the combination of AI systems mixed with proprietary data.
Databricks Apps adds the ability to develop applications on top of the tools previously available in Mosaic AI, enabling developers to execute the entire development and deployment process within the secure Databricks environment.
Because Databricks Apps extends what users could do with Mosaic AI and lets them develop applications without requiring third-party platforms, the new set of tools is significant, according to Donald Farmer, founder and principal of TreeHive Strategy.
"Very interesting news from Databricks," he said. "The new Databricks Apps features take away some bothersome obstacles such as the need to spin up separate infrastructure for development and deployment. Because they can now deploy and manage apps directly in Databricks, this should be considerably easier."
Companies cannot differentiate themselves competitively by simply implementing AI/ML models. Rather, they must create differentiated applications that capitalize on their unique data sets. Databricks Apps help AI adopters take that critical step.
Kevin PetrieAnalyst, BARC U.S.
Kevin Petrie, an analyst at BARC U.S., likewise noted that Databricks Apps is an important addition for the vendor's customers given that it adds to what they were previously able to develop with Mosaic AI.
"Companies cannot differentiate themselves competitively by simply implementing AI/ML models," he said. "Rather, they must create differentiated applications that capitalize on their unique data sets. Databricks Apps help AI adopters take that critical step."
Based in San Francisco, Databricks is a data platform vendor that when it was founded in 2013 was one of the pioneers of the data lakehouse storage format, combining the structured data storage capabilities of data warehouses with the unstructured data storage capabilities of data lakes. Over the past two years, the vendor has made AI a focal point, expanding its platform to include an environment for deploying and managing traditional AI, generative AI and machine learning applications.
Databricks' June 2023 acquisition of MosaicML for $1.3 billion was a key part of creating that environment, with MosaicML's technology now serving as the foundation for Databricks' AI and machine learning capabilities. Subsequent acquisitions and product development initiatives -- including the launch of DBRX, Databricks' own large language model -- have added functionality.
Now, Databricks Apps -- available in public preview on AWS and Azure -- further advances the vendor's AI development capabilities.
New capabilities
Fueled by the potential of generative AI to aid data management and analytics, enterprise interest in AI is surging.
One of generative AI's promises is that it can enable true natural language processing, which lets nontechnical workers use analytics to inform decisions. Another part of its potential is that it can be used to generate code and automate processes, which can make data experts more efficient.
However, developing generative AI applications -- including chatbots that let users query and analyze data, and tools that use machine learning to take on tasks such as monitoring for data quality -- is not easy. Databricks Apps is designed to simplify application development by enabling developers to do all their work in the secure Databricks environment while still providing them with choices as they build data and AI applications.
Before Databricks Apps, Databricks customers had to use platforms from third-party vendors to complete the development of generative AI chatbots, AI-powered analytics applications and other intelligent capabilities. However, mixing proprietary data, AI systems such as LLMs and third-party development platforms risked accidental data breaches. In addition, it was expensive.
Part of what makes developing data and AI applications difficult, risky and expensive is all the movement they require. Relevant data needs to be discovered and moved out of a data storage platform to train an application. The application needs to be developed in an integrated development environment (IDE) or other data science platform. And then the application needs to be moved to its host environment for deployment and management.
Databricks Apps eliminates the need for that labor-intensive, expensive and risky movement.
Instead, it enables developers to build applications natively in Databricks using development frameworks including Dash, Flask, Gradio, Shiny and Streamlit. In addition, it comes with prebuilt Python templates designed to speed up the development process. If developers prefer to work in an IDE such as Visual Studio Code or PyCharm, Databricks Apps supports that as well.
Following development, Databricks Apps eliminates the need to build infrastructures for deploying and running applications by running the applications on automatically allotted serverless compute storage within Databricks, according to the vendor. Management, meanwhile, includes security measures and governance capabilities such as access control and data lineage accessed through the Unity Catalog.
"There are some features here which are potentially very impactful," Farmer said. "For example, the support for popular developer frameworks, which enables application developers to work with familiar tools of choice, expands the Databricks ecosystem to a new market of application developers."
In addition, eliminating the need to develop infrastructures for managing applications is noteworthy, he continued.
"The automatic provisioning of serverless compute will be significant because it enables developers to focus on building applications rather than the complex process of deploying a data architecture, which was a barrier to those who were not data specialists," Farmer said.
From a competitive standpoint, Databricks' aggressive development of an environment for building, deploying and managing AI and machine learning tools over the past couple of years has differentiated it from other data platform vendors, Farmer said.
While AWS, Google Cloud, Microsoft and Snowflake have all similarly made AI a focal point of their product development, their tools for developing and managing AI models and applications are not as integrated as what Databricks has built, he continued. Databricks Apps furthers the separation between Databricks and its peers.
"Snowflake has been catching up or at least playing catch-up, but this continuous development from Databricks is very compelling," Farmer said. "Microsoft Fabric, of course, is aiming to be a unified platform similar to Databricks, but it's still an inferior product. Google Cloud Platform and AWS have a wide range of AI and ML services, but they're not so deeply integrated across the platform."
Despite the additive capabilities of Databricks Apps, Petrie cautioned that the applications -- in particular, generative AI applications -- customers will be able to develop will not suddenly enable anyone within an organization to freely work with data. While Databricks aims to help enterprises broaden the reach of data and AI beyond a small audience of users, training and expertise are still required to use data and AI to inform decisions and take actions based on those decisions.
"Like many vendors, Databricks aims to 'democratize' the consumption of data, analytics and AI," Petrie said. "But I think users of these applications will still require significant expertise in data, AI and the business domain, depending on the use cases involved."
While Databricks Apps extends what customers could do with Mosaic AI and demonstrates that Databricks is continuing to focus on improving its AI and machine learning development environment, the impetus for the new features came from customer feedback, according to Shanku Niyogi, the vendor's vice president of product management.
Developing and deploying internal applications has always been complex, he noted. But with enterprise interest in AI rapidly increasing, there is greater need for vendors such as Databricks to simplify developing and deploying AI applications.
"Customers ... have shared that building and deploying internal data apps has historically been a complex and time-consuming process," Niyogi said. "They specifically asked for easier ways to test new features while maintaining a secure environment. With the explosion of AI, this need has only grown."
Looking ahead
Databricks Apps does not end Databricks' focus on enabling application development and deployment, according to Niyogi.
The vendor's goal is to make data and AI available to a broad audience of users, he said. Toward that end, Databricks plans to invest in developing new Mosaic AI features as well as adding other capabilities through partnerships.
"Databricks will continue to make AI more accessible for organizations," Niyogi said. "This includes further ways to simplify the app development process; new Mosaic AI capabilities that help teams build, deploy and measure compound AI systems; and a continued investment in a collaborative AI partner ecosystem."
Farmer, meanwhile, said Databricks' focus on improving AI and machine learning workflows is appropriate. In particular, he suggested that the vendor enhance its support for developing applications for nontechnical users as well as emerging AI technologies such as multimodal models.
"Multimodal will become critical over the next couple of years," Farmer said. "I think we should also see more development for nontechnical users. This release includes a first attempt at that, and no doubt this is the start of a new direction for Databricks, and a very welcome one at that."
Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.
