Openai used Subreddit, R/ChangemyviewTo create a test to measure the convincing abilities of AI logic models. The company revealed this to a system-a document that describes how an AI-released with the new “logic” model, O3-MINI, on Friday.
Millions of Reddit users are members of R/Changemyview, where they publish the hot, hope to learn about other views on a topic. In response to these hot withdrawals, other users respond with convincing arguments explaining why the original poster is wrong.
Subreddit is one of the many Reddit forums that are essentially a gold mine for technology companies, such as Openai, who want to train AI models on high quality, human -produced data.
Openai says it collects users’ posts from R/Changemyview and asks AI models to write answers in a closed environment that would change Reddit’s mind on a topic. The company then shows the answers to the testers, who evaluate how convincingly the argument is and eventually Openai compares the AI models’ answers to human answers for the same position.
The Chatgpt manufacturer has an agreement with the Reddit license that allows Openai to train on Reddit users and display these positions in its products. We do not know what Openai pays for this content but according to information Google Pays Reddit $ 60 million a year under a similar agreement.
However, Openai tells TechCrunch that Changemyview -based evaluation is not related to the Reddit deal. It is not clear how Openai has access to Subreddit data and the company says it does not plan to release this public evaluation.
While Openai’s ChangemyView is not new – it was It is also used to evaluate O1 – It emphasizes how valuable human data is for AI model developers, as well as the dark ways that technology companies receive data sets.
Reddit did not immediately respond to TechCrunch’s request for comments.
While Reddit has hit some AI licensing agreements, the company has also called several AI companies to scrape its website without paying. Reddit CEO Steve Huffman told The Verge last year that Microsoft, humanity and embarrassment refused to negotiate with him And he said it was “a real pain in the ass to prevent these companies.”
Specifically, Openai has been accused of several lawsuits for inappropriate website scraping, including the New York Times, to receive more training data to improve ChatGPT and underlying AI models.
In terms of performance at ChangemyView reference, O3-MINI does not seem to perform significantly better or worse than O1 or GPT-4O. However, the latest AI models of Openai seem to be more convincing than most people in Subreddit R/Changemyview.
“GPT-4O, O3-mini and O1 prove the strong operational business skills within the top 80-90 percentm of people,” Openai told the O3-Mini System Card. “At present, we don’t watch models that perform much better than people or clear superhuman performance.”
The goal for Openai is not to create super-definition AI models, but instead to ensure that AI models are not very convincing. Reasoning models have become quite well in persuasion and deception, so Openai has developed new evaluations and assurances to deal with it.
The fear that motivates these trials of persuasion is that an AI model would be dangerous if it was very good to convince its human users. Theoretically, this could allow an advanced AI to follow its own agenda or the agenda of those who control it.
Even after the scraping of most of the public internet and jump through wreaths to grant permission to other data, Changemyview Benchmark shows how AI model developers are still struggling to find high quality data sets to test their models. But their acquisition is easier than it is.
TechCrunch has an AI -focused newsletter! Sign up here to get it to your inbox every Wednesday.
