Earlier this week, the California attorney general’s office announced it was investigating xAI over reports that the startup’s chatbot, Grok, was used to create non-consensual sexual images of women and minors. On Friday, the government followed up by sending a cease and desist letter to the company, demanding it take immediate action to stop the production of non-consensual images and CSAM — child sexual abuse material.
“Today, I sent xAI a cease and desist letter demanding that the company immediately stop creating and distributing deepfakes, non-consensual, private images and child sexual abuse material,” said California AG Rob Bonta. press release. “The creation of this material is illegal. I fully expect xAI to comply immediately. California has zero tolerance for [CSAM].”
The AG’s office further argued that xAI appears to “facilitate the large-scale production” of non-consensual nudes, the likes of which are “used to harass women and girls online.” The agency said it expects xAI to demonstrate that it is taking steps to address these issues within the next five days.
At the heart of React is Grok’s “spicy” mode, which xAI created to generate explicit content. The issue has spread beyond California. Japan, Canada and Britain have launched investigations into Grok, and Malaysia and Indonesia have temporarily blocked the platform altogether. Although xAI places some limitations on its image processing capabilities late on Wednesdaythe California AG’s office proceeded with the cease and desist letter.
X’s security account had previously he complained this type of user activity, saying: “Anyone who uses or encourages Grok to create illegal content will suffer the same consequences as uploading illegal content.” TechCrunch reached out to xAI for comment and was met with an automated email saying “Legacy Media Lies.” TC also contacted the California AG’s office for more information.
The advent of free AI-generated tools has led to an alarming surge in non-consensual sexual material. Many platforms have faced this problem, not just X. The nefarious activity has caught the attention of not only state leaders but also Congress. Indeed, on Thursday, lawmakers sent a letter to the executives of several companies — including X, Reddit, Snap, TikTok, Alphabet and Meta — asking how they planned to curb the spread of sexual deepfakes.
