It doesn’t take much to figure out that GenAI is spewing falsehoods and untruths.
This past week provided an example, with chatbots from Microsoft and Google announcing a Super Bowl winner before the game even started. The real problems start, though, when GenAI’s hallucinations become harmful — approving torment, booster ethnic and racial stereotypes and persuasive writing about conspiracy theories.
A growing number of vendors, from incumbents like Nvidia and Salesforce to startups like CalypsoAI, offer products that they claim can mitigate unwanted, toxic content from GenAI. But they are black boxes. Without testing each one individually, it’s impossible to know how these anti-hallucination products compare — and whether they really live up to the claims.
Shreya Rajpal saw this as an important problem — and started a company, AI Guardrailsto try to solve it.
“Most organizations … are struggling with the same set of problems around responsible AI application development and struggling to figure out what the best and most effective solution is,” Rajpal told TechCrunch in an email interview. “They often end up reinventing the wheel in terms of managing the set of risks that are important to them.”
According to Rajpal, research suggests that complexity—and by extension risk—is a top barrier preventing organizations from embracing GenAI.
Recent voting by Intel subsidiary Cnvrg.io found that compliance and privacy, reliability, high implementation costs and a lack of technical skills were concerns shared by about a quarter of companies implementing GenAI applications. In separate overview by Riskonnect, a provider of risk management software, more than half of executives said they were concerned that employees were making decisions based on inaccurate information from GenAI tools.
Rajpal, who previously worked at self-driving startup Drive.ai and, following Apple’s acquisition of Drive.ai, on Apple’s special projects team, co-founded Guardrails with Diego Oppenheimer, Safeer Mohiuddin and Zayd Simjee. Oppenheimer previously led Algorithmia, a machine learning operations platform, while Mohiuddin and Simjee held leading technology and engineering roles at AWS.
In some ways, what Guardrails offers isn’t that different from what’s already on the market. The startup’s platform acts as a wrapper around GenAI models, especially open source and proprietary text generation models (eg OpenAI’s GPT-4), to make these models seemingly more reliable, trustworthy and secure.
Image Credits: AI Guardrails
But where Guardrails differs is its open source business model — the platform’s code base is available on GitHub, free to use — and its crowdsourced approach.
Through a marketplace called Guardrails Hub, Guardrails allows developers to submit modules called “validators” that interrogate GenAI models for certain behavior, compliance and performance metrics. Validators can be developed, reused, and reused by other Guardrails developers and customers, serving as building blocks for custom GenAI model supervision solutions.
“With the Hub, our goal is to create an open forum to share knowledge and find the most effective way [further] AI adoption — but also to build a set of reusable guardrails that any organization can adopt,” Rajpal said.
The validation tools in Guardrails Hub range from simple rules-based checks to algorithms to detect and mitigate problems in models. There are about 50 currently, ranging from hallucinations and policy violation detectors to filters for proprietary information and insecure code.
“Most companies will do broad, uniform checks for profanity, personally identifiable information and so on,” Rajpal said. “However, there is no single, universal definition of what constitutes acceptable use for a particular organization and group. There are organization-specific risks that need to be monitored — for example, communication policies between organizations are different. With the Hub, we’re enabling users to use the solutions we provide out of the box or use them to get a powerful starting solution that they can further customize for their specific needs.”
A node for model guardrails is an interesting idea. But the skeptic in me wonders if developers will bother contributing to a platform—and a nascent one at that—without the promise of some form of compensation.
Rajpal is optimistic that it will, if for no other reason than recognition — and to selflessly help the industry build toward “safer” GenAI.
“The Hub allows developers to see the types of risks other businesses face and the guardrails they put in place to resolve and mitigate those risks,” he added. “Validators are an open source implementation of these guardrails that organizations can apply to their use cases.”
Guardrails AI, which doesn’t yet charge for any services or software, recently raised $7.5 million in a seed round led by Zetta Venture Partners with participation from Factory, Pear VC, Bloomberg Beta, GitHub Fund and corners like renowned expert AI Ian Goodfellow. Rajpal says the proceeds will go toward expanding Guardrails’ six-person team and toward additional open source projects.
“We’re talking to so many people—enterprises, small startups, and individual developers—who are stuck on being able to ship GenAI applications because of a lack of assurance and risk mitigation,” he continued. “This is a new problem that didn’t exist at this scale because of the emergence of ChatGPT and foundation models everywhere. We want to be the ones to solve this problem.”
