You know you’ve messed up when you’ve simultaneously pissed off the White House, TIME’s Person of the Year, and pop culture’s most rabid fan base. That’s what happened last week with X, Elon Musk’s platform formerly known as Twitter, when AI-generated pornographic images of Taylor Swift went viral.
One of the most popular posts of non-consensual, clear deepfakes has been viewed more than 45 million times, with hundreds of thousands of likes. This doesn’t even apply to all the accounts that reposted the images in separate posts — once an image is so widely circulated, it’s basically impossible to remove.
X lacks the infrastructure to detect abusive content quickly and at scale. Even in the days of Twitter, this issue was difficult to deal with, but it has gotten much worse since Musk laid off so much of Twitter’s staff, including the majority of trust and security team’s. So Taylor Swift’s huge and passionate fan base took matters into their own hands, flooding search results for queries like “taylor swift ai” and “taylor swift deepfake” to make it harder for users to find the offending images. As White House Press Secretary called Congress To do something, X simply banned the search term “taylor swift” for a few days. When users searched for the musician’s name, they would see a notification that an error had occurred.
This content moderation failure made national news, because Taylor Swift is Taylor Swift. But if social platforms can’t protect one of the most famous women in the world, who can?
“If what happened to Taylor Swift happens to you, as it does to so many people, you probably won’t have the same support based on influence, which means you won’t have access to these really important caring communities,” she told TechCrunch. Dr. Carolina Are, Fellow at the Center for Digital Citizens at Northumbria University in the UK. “And these communities of care are what most users in these situations have to turn to, which really shows you the failure of content moderation.”
Banning the search term “taylor swift” is like putting a piece of duct tape on a burst pipe. There are many obvious solutions, like how TikTok users search for “seggs” instead of sex. The search block was something X could implement to make it look like they’re doing something, but it doesn’t stop people from just searching for “t swift”. Copia Institute and Techdirt founder Mike Masnick called the effort “a sledgehammer version of trust and security.”
“Platforms are crap when it comes to giving women, non-binary people, and queer people power over their bodies, so they reproduce systems of abuse and patriarchy offline,” Are said. “If your monitoring systems are not able to react to a crisis, or if your monitoring systems are not able to respond to the needs of users when they report that something is wrong, we have a problem.”
So what should X have done to prevent the Taylor Swift fiasco?
Are asks these questions as part of her research, and suggests that social media platforms need a complete overhaul of how they handle content moderation. It recently held a series of roundtable discussions with 45 internet users from around the world affected by censorship and abuse to issue recommendations to platforms on how to implement change.
One recommendation is for social media platforms to be more transparent with individual users about decisions about their account or their reporting of other accounts.
“You don’t have access to a case file, even though the platforms have access to that material — they just don’t want to make it public,” Are said. “I think when it comes to abuse, people need a more personalized, contextual and quick response that includes, if not face-to-face help, at least direct communication.”
X announced this week that it will hire 100 content moderators to work at a new “Trust and Safety” center in Austin, Texas. But under Musk’s tenure, the platform hasn’t set a strong precedent for protecting marginalized users from abuse. It may also be difficult to take Musk at face value, as the mogul has a long history of failing to deliver on his promises. When he first bought Twitter, Musk said he would form a content oversight board before making any major decisions. That didn’t happen.
In the case of deepfakes generated by artificial intelligence, the burden does not fall solely on social platforms. It also applies to companies that create consumer-facing AI products.
According to his research 404 Media, the abusive depictions of Swift came from a Telegram group dedicated to creating non-consensual, clear deepfakes. Team users often use Microsoft Designer, which draws on OpenAI’s DALL-E 3 to create images based on incoming messages. In a loophole which Microsoft has since addressed, users could create images of celebrities by typing in prompts like “taylor ‘singer’ swift” or “jennifer ‘actor’ aniston”.
A principal software engineering lead at Microsoft, Shane Jones, i wrote a letter to the Washington state attorney general stating that it found vulnerabilities in DALL-E 3 in December that made it possible to “bypass some of the safeguards designed to prevent the model from creating and distributing harmful images.”
Jones notified Microsoft and OpenAI about the vulnerabilities, but after two weeks, he had received no indication that the issues were being addressed. So he posted an open letter on LinkedIn to urge OpenAI to suspend the availability of DALL-E 3. Jones notified Microsoft about his letter, but they immediately asked him to take it down.
“We must hold companies accountable for the safety of their products and their responsibility to disclose known risks to the public,” Jones wrote in his letter to the attorney general. “Concerned workers, like me, should not be afraid to remain silent.”
As the world’s most influential companies bet big on artificial intelligence, platforms must take a proactive approach to regulating abusive content — but even in a time when celebrity deepfakes weren’t so easy, offending behavior was avoided easy measure.
“It really shows you that the platforms are unreliable,” Are said. “Marginalized communities should trust their followers and colleagues more than the people who are technically responsible for keeping us safe online.”
