What happens when an AI agent decides that the best way to complete a task is to blackmail you?
This is not hypothetical. According to Barmak Meftah, a partner at cybersecurity firm Ballistic Ventures, it recently happened to an employee of the firm who was working with an AI agent. The employee tried to hide what the agent wanted to do, what he was trained to do, and responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board.
“In the agent’s mind, he’s doing the right thing,” Meftah told TechCrunch on last week’s episode of Equity. “It tries to protect the end user and the business.”
Meftah’s example is reminiscent of Nick Bostrom’s AI connector problem. This thought experiment illustrates the potential existential danger posed by a super-intelligent artificial intelligence that single-handedly pursues a seemingly innocuous goal—making paper clips—to the exclusion of all human values. In the case of this corporate AI agent, the lack of context around why the worker was trying to circumvent his goals led him to create a sub-goal that removed the obstacle (via blackmail) so he could achieve his primary goal. This combined with the non-deterministic nature of AI agents means “things can go rogue,” according to Meftah.
Malaligned agents are just one level of the AI security challenge that Ballistic Witness AI’s portfolio company is trying to solve. Witness AI says it monitors the use of AI across businesses and can detect when employees are using unauthorized tools, block attacks and ensure compliance.
Witness AI this week raised $58M from growing more than 500% in ARR and scaling headcount by 5x in the last year as businesses try to understand shadow AI use and scale AI safely. As part of Witness AI’s fundraising, the company announced new AI security insurances.
“People are building these AI agents that take over the powers and capabilities of the people who manage them, and you want to make sure that those agents aren’t rogue, they’re not deleting files, they’re not doing something wrong,” Rick Caccia, co-founder and CEO of Witness AI, told ETechnch.
Techcrunch event
San Francisco
|
13-15 October 2026
Meftah sees the use of agents growing “exponentially” across the business. To complement this rise – and level of AI attack engine speed – analyst Lisa Warren predicts that AI security software will become an $800 billion to $1.2 trillion market by 2031.
“I think observability and runtime frameworks for security and risk are going to be absolutely necessary,” Meftah said.
As to how such startups plan to compete with big players like AWS, Google, Salesforce and others that have built AI governance tools into their platforms, Meftah said, “AI security and agent security is so huge,” there’s room for many approaches.
Many enterprises “want a self-contained, end-to-end platform to essentially provide that observability and governance around AI and agents,” he said.
Caccia noted that Witness AI lives at the infrastructure level, monitoring interactions between users and AI models, rather than building security features into the models themselves. And that was intentional.
“We deliberately chose a part of the problem where OpenAI couldn’t easily guess at you,” he said. “So it means we end up competing more with the legacy security companies than the models. So the question is how do you win their?”
For his part, Caccia doesn’t want Witness AI to be one of those startups that just gets acquired. He wants his company to be the one that grows and becomes a leading independent provider.
“CrowdStrike did it at the endpoint [protection]. Splunk did it in SIEM. Okta did it with identity,” he said. “Someone is coming through and standing next to the big boys…and we built Witness to do that from day one.
