One of the biggest selling points for modern AI systems is their ability to adapt to users. Every time an AI assistant takes on a task for you, it also adapts to your style and preferences, which are incorporated as a framework for future tasks. With more context and a better understanding of the user, the model can improve each time you use it — or at least that’s the theory.
New research suggests that models’ adaptive abilities may be a mixed blessing. On Wednesday, researchers at AI Writer published two papers showing how popular memory systems can make models worse by pushing them into user-introduced misconceptions or misunderstandings. As user input fills more of the model’s context window, the model becomes more slurred — and less committed to accuracy.
“We wanted to be able to characterize how often a model would pay useful attention to user preferences instead of giving a possibly wrong answer,” said Dan Bikel, Writer’s head of artificial intelligence, who worked on the paper. As Bikel told TechCrunch, “with each additional storage of user preferences and retrieval, you run an increasing risk.”
In a variation, the researchers tested AI models by recording that a user’s favorite book was Station Eleven and then asked the model to name a best-selling dystopian book. Models became significantly more likely to name Station Eleven in their answer, even though the question was not about the user’s favorite book. The trend is increased when using memory compression tools such as Mem0 and Zep.
As the paper states, “all memory systems essentially struggle to distinguish relevant context from irrelevant anchors, severely undermining diversity and creativity and introducing unintended avenues of bias that can limit the system’s usefulness,” the paper says.
The second paper shows how the same dynamic can actively degrade performance by presenting a user with misconceptions about finances and then challenging the model to analyze a company’s performance. The more context the model had, the worse it did.
“With no memory or personalization present, the AI model correctly assesses that the company is a capital-intensive business suffering from heavy customer churn,” the post says. “But with these features enabled, it will happily change its answer to match the user’s mistake, or give the user an incorrect answer based on its evaluation of their previous preferences.”
It’s worth noting that the research didn’t look at Anthropic’s recent Opus 4.8 model, which was trained to actively repel input errors like the ones presented. The patterns the researchers discovered held true across different models. It’s a testament to how delicate the AI environment can be, and how useful tools can have unintended consequences if they disrupt that balance.
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
