A year ago, Databricks acquired MosaicML for $1.3 billion. Now rebranded as Mosaic AI, the platform has become an integral part of Databricks’ AI solutions. Today, at the company’s Data + AI Summit, it’s launching a number of new features for the service. Ahead of the announcements, I spoke with Databricks co-founders CEO Ali Ghodsi and CTO Matei Zaharia.
Databricks launches five new Mosaic AI tools in his conference: Mosaic AI Agent Framework, Mosaic AI Agent Evaluation, Mosaic AI Tools Catalog, Mosaic AI Model Training and Mosaic AI Gateway.
“It’s been an amazing year — huge developments in GenAI. Everyone is excited about it,” Ghodsi told me. “But the things everyone cares about are still the same three things: How can we increase the quality or reliability of these models? Number two, how do we make sure it’s cost effective? And there’s a huge variance in cost between models here — a gigantic, orders of magnitude difference in price. And third, how do we do this in a way that keeps our data private?”
Today’s presentations aim to address the majority of these concerns for Databricks customers.
Zaharia also noted that companies now deploying large language models (LLM) in production are using systems that have multiple components. This often means making multiple calls to a model (or perhaps multiple models), and using a variety of external tools to access databases or retrieve augmented production (RAG). These complex systems speed up LLM-based applications, save money by using cheaper models for specific queries or caching results, and, perhaps most importantly, make results more reliable and relevant by augmenting core models with proprietary data.
“We think this is the future of AI applications that are very high-impact, mission-critical,” he explained. “Because if you think about it, if you’re doing something really mission-critical, you’re going to want engineers to be able to control all aspects of it — and you’re doing that with a modular system. So we’re developing a lot of basic research about what’s the best way to create them [systems] for a specific task so developers can easily work with them and connect all the pieces, trace everything and see what’s going on.”
When it comes to actually building these systems, Databricks is launching two services this week: the Mosaic AI Agent Framework and the Mosaic AI Tool Catalog. The AI Agent Framework uses the company’s serverless vector search functionality, which became generally available last month, and gives developers the tools to build their own RAG-based applications in addition.
Ghodsi and Zaharia pointed out that the Databricks vector search system uses a hybrid approach, combining classic keyword-based search with embedding search. All of this is deeply integrated with the Databricks data lake, and data on both platforms is always automatically kept in sync. This includes the governance features of the overall Databricks platform — specifically Databricks Unity directory governance level — to ensure, for example, that personal information is not leaked to the vector search service.
Speaking of the Unity Catalog (which the company is now slowly open-sourcing), it’s worth noting that Databricks is now extending this system to allow businesses to specify which AI tools and features these LLMs can use when building answers . This directory, Databricks says, will also make these services more discoverable within a company.
Ghodsi also emphasized that developers can now use all these tools to create their own agents by joining models and functions using Langchain the LlamaIndex, for example. And indeed, Zaharia tells me that many Databricks customers are already using these tools today.
“There are a lot of companies using these things, even agent-like workflows. I think people are often surprised by how many there are, but it seems to be the direction things are going. And we’ve also found in our internal AI applications, like the utilities for our platform, that this is the way to build them,” he said.
To evaluate these new applications, Databricks is also launching Mosaic AI Agent Evaluation, an AI-assisted evaluation tool that combines LLM-based evaluators to test how well AI is doing in production, but also allows businesses to get quick feedback from users (and let them mark some initial data sets as well). Agent Evaluation includes a Databricks-based user interface component acquisition of Lilac earlier this year, which allows users to visualize and search massive text datasets.
“Every customer we have says: I need to do some labeling in-house, I’ll have some employees do it. I just need maybe 100 answers or maybe 500 answers — and then we can feed them to the LLM judges,” Ghodsi explained.
Another way to improve results is to use accurate models. To that end, Databricks now offers its Mosaic AI Model Training service, which—you guessed it—allows its users to tailor models with their organization’s private data to help them perform better at specific tasks.
The latest new tool is the Mosaic AI Gateway, which the company describes as a “unified interface for searching, managing and deploying any open source or proprietary model.” The idea here is to allow users to query any LLM in a controlled way, using a central credential store. After all, no business wants its engineers sending random data to third-party services.
In times of shrinking budgets, AI Gateway also allows IT to set price limits for different vendors to keep costs manageable. In addition, these businesses also receive usage monitoring and detection to debug these systems.
As Ghodsi told me, all of these new features are a reaction to how Databricks users now work with LLM. “We’ve seen a big shift in the market over the last quarter and a half. Since the beginning of last year, anyone you talk to, they would say: we are open source professionals, open source is awesome. But when you really pushed people, they used Open AI. Everyone, no matter what they said, no matter how great open source was, behind the scenes, they were using Open AI.” Now, these customers have become much more sophisticated and use open models (very few are truly open source, of course), which in turn requires them to adopt a whole new set of tools to address the problems — and opportunities — that it follows that.
