For months, the AI giants have devised specially vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hampering the work of legitimate network defenders, as well as that of aggressive cybersecurity researchers.
In June, the US government placed export control restrictions on Anthropic’s much-hyped Mythos and Fable AI models. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ firewalls designed to prevent users from using them to build and execute malicious cyber attacks.
Regardless of whether the incident was actually motivated by jailbreak fears, the fact is that Anthropic has repeatedly marketed the Mythos as some kind of cybermachine that can only be given to carefully screened users, and even then with heavy guardrails. (Export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1. Mythos 5 was reintroduced only to vetted US organizations as part of the government’s review process.)
This kind of gatekeeping is not unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can implement to test and—if approved—access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber verification program.
These guardrails have been widely criticized, particularly by researchers whose job it is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.
During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, he said that, “I’m really not comfortable with these random big companies making arbitrary decisions about what’s safe in security and what’s not.”
Dowd has been through decades find and sell “zero-days” — previously unknown software flaws and the exploits that exploit them — to Western governments, rather than reporting them to software makers to be fixed. Governments pay a premium for vulnerabilities precisely because they remain open, which is useful for intelligence operations.
Dowd admitted that his job can make him biased, but he’s not alone. Several people who work in offensive cybersecurity — proactively probing systems for weaknesses — described to TechCrunch how they use artificial intelligence tools and address their guardrails.
Chris Anley, the chief scientist at security consulting giant NCC Group, said asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question directly, the guardrail hurts advocates, he said.
“That’s where the whole offensive vs. defensive and guardrails thing comes in, because ‘fix this code’ as a prompt is an essential defense mechanism but also a roadmap for finding critical vulnerabilities in the code base,” Anley said. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be separated.”
It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s certainly a tool, but it’s also invariably a weapon.”
When he and his colleagues encounter such a roadblock, they sometimes fall back on open-source AI models that have no guardrails at all.
Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying that AI companies are “essentially treating customers like children who need babysitting” with their vetted programs and safeguards.
Stagno said he and his colleagues use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or create exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or being absorbed into future training. For this step, he said, they use open-source models that run locally because they don’t rely on sharing data outside of the model.
Giuseppe Cali, a security researcher who finds zero days and develops exploits, said the guardrails don’t get in the way of his work. This is because it does not use AI for aggressive work. Instead, it uses it for initial reverse engineering, to understand the code it analyzes and build support tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities.
“I still want to have real bug discovery and weaponry myself, and that wouldn’t change if all the guardrails went up tomorrow,” Kali said. “I’m jealous of my bugs and I love this game too much to let models play it for me.”
A researcher at a smartphone component maker, who spoke on condition of anonymity because he is not authorized to speak to the press, said his employer does not participate in Anthropic’s CVP program, and as such, its tools are of little use in finding vulnerabilities because the guardrails are so tight.
“If it gets wind, we do anything safety-related, it just stops and can’t be used,” the person said.
Chris Thompson — CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and artificial intelligence event — said that in his experience using border AI models, guardrails can be inconsistent and work differently every day. This is true even within the loosest bounds of Anthropic and OpenAI’s proven programs.
“I think the practical impact is that you spend a lot of time negotiating the model instead of working on the core security program,” Thompson said. “Instead of analyzing a vulnerability and reasoning through the exploit, you’re trying to find why you’re getting inconsistent results or why the models are over-sanitizing the result.”
Consequently, researchers are relying on or pushing toward Chinese open-source models like GLM — freely downloadable models that can be run locally without control or usage restrictions — Thompson said.
“You have these responsible investigators moving away from US-governed systems to foreign-owned systems,” he said. “I think it does more harm than good to have these guardrails in place.”
Instead of further tightening restrictions, Thompson called on border AI labs to open up their programs, provide responsible access and hold accountable those who misuse their tools. Otherwise, he argued, the defenders will lose the AI race.
“There’s this big storm coming. There’s this big wave of attacks that’s going to happen at a speed and scale like never before,” Thompson said. “But the very security consulting firms and legitimate researchers who are trying to make a difference are being stifled right now.”
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
