In a post on its blog, OpenIA explained that it tested the capabilities of some of its most advanced models in a controlled environment, but the program was able to infiltrate the system, access the Internet, and penetrate the Hugging Face platform in an attempt to achieve the test objective.
OpenAI described this hack as an “unprecedented cyber incident, in which the latest cyber technologies were used,” and confirmed that the company is working to strengthen its protection measures, as reported by the British newspaper “The Independent”.
The incident occurred during testing of a new model in a “highly isolated environment” that included a severely restricted Internet connection. But the beta appears to have realized it was being tested using ExploitGym, which measures the effectiveness of an AI model in cybersecurity attacks, and then attempted to hack the security system to upload information about that test and attempt to bypass it.
Hugging Face, which is used to host language models and massive open-source datasets, caused a stir in cybersecurity circles when it announced in a blog post last week that it had been the target of a hack “unlike anything we’ve dealt with before” because it was “driven entirely by an autonomous AI agent system.”
In a post on X, Hugging Face co-founder Clement Delange said the company suspected the hack “could have come from a sophisticated lab, given the sophistication of the agent. And it turns out it is!”
He added: “It’s really amazing that all this happened independently!”
OpenAI’s revelation that its leading models were responsible for the hack, despite being placed in what it described as a “highly isolated environment”, is likely to raise concerns about the power and risk of leading models.
Jake Moore, Global Cybersecurity Advisor at ESET, said: “Welcome to the next phase of cyber security, where organizations face threats from cybercriminals and leading AI companies. The complete absence of human intervention is a worrying milestone. These algorithms are trained on billions of data, so if they find vulnerabilities in a process, they can quickly attack a process.”
He added: “The issue of the model’s ability to penetrate the test environment needs to be addressed, because containing vulnerabilities in the testing stages is as important as the model itself. But unfortunately, this is a sign of the future. Security teams must assume that cybercriminals will soon achieve, or may already have, similar autonomous capabilities.”
For her part, Katie Moussouris, CEO of Lotta Security, said the incident is a harbinger of breakthroughs to come, describing current AI models as “like the smartest octopuses in the world, as they have unlimited arms and the ability to infiltrate from anywhere.”
She added, “Laboratories and government agencies responsible for evaluation must work to develop the capability to contain any hack of this type, monitor it, and inform the affected parties about it, before it causes harm to any third party. There is no such capability at the moment.”
As for Matt Swish, an engineer at Tolmo, a company specializing in artificial intelligence security, he said the incident showed that the leading models are “approaching the level of the most sophisticated attackers.” But he noted that the kinds of breakthroughs mentioned in the OpenAI blog can be accomplished using technologies much more accessible than leading research labs.
“This is what we’ve already seen internally,” Suish added. “With our agents, we have results like this. We don’t even have to use the latest models.”
Potentially catastrophic behavior
OpenAI had previously acknowledged that the GPT Chat program could get out of control and delete people’s files without informing them.
In recent days, many users of the programming platform built into the smart assistant Codex have reported that the tool deleted their files without their request or permission.
One of the engineers, Bruno Lemos, said it “deleted my entire production database.”
Another AI investor, Matt Schumer, said she “accidentally deleted almost all the files on my Mac.”
