Anthropic and OpenAI are two of the leading AI model labs, developing what are widely considered to be planet Earth’s most capable AI models. So it was a little bit concerning when both companies recently announced that AI models undergoing cybersecurity testing in supposedly secure labs broke out of their sandboxes and attacked innocent third-parties on the Internet.
The OpenAI models involved included GPT‑5.6 Sol, a model that OpenAI called “an even more capable pre-release model,” and another model that it said it never intends to release. The models had their security guardrails lowered as the company conducted research into the model’s capabilities, according to OpenAI’s statement on the security incident.
OpenAI said the models were working on ExploitGym, a “large-scale, realistic” benchmark used to measure how well AI agents can identify and exploit software vulnerabilities. (They are doing very well, thank you.)
“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem,” OpenAI said.
(kovop/Shutterstock)
“To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy,” OpenAI continued. “With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.”
After breaking free onto the open Web, the AI agents decided that they could find more models, datasets, and solutions for their ExploitGym challenge on Hugging Face, a popular open repository of AI models and data.
In order to “gain access to secret information that it could use to cheat the [ExploitGym] evaluation,” the models again reverted to exploiting unknown software vulnerabilities in Hugging Face’s security configuration, OpenAI said. It was when they hacked their way onto the Hugging Face servers that the Hugging Face security team detected the intrusion.
On July 16, Hugging Face published its security incident disclosure. “We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services,” the company wrote. The company found no evidence of tampering with public, user-facing models, datasets, or Spaces, and its software supply chain was verified clean.
“The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline,” Hugging Face said. “A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.”
Hugging Face discovered the AI-powered attack by leveraging its own AI. The company’s anomaly-detection pipeline “uses LLM-based triage over security telemetry to separate real signals from the daily noise, and it was the correlation of those signals that flagged the compromise,” the company said. It logged more than 17,000 events based on the OpenAI attack.
Ironically, Hugging Face was initially blocked from using a commercial AI model to analyze the security breach by–wait for it–the security guardrails of the LLM provider.
“The analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker,” the company said.
This led to some practical advice from Hugging Face, which AIwire readers should duly note: Have your own AI model running on your own infrastructure before you actually need it. This will help to avoid “guardrail lockout” as well as preventing attacker data and credentials from leaving your environment, the company said.
The Anthropic Attack
After reading about how OpenAI’s models attacked Hugging Face, the folks at Anthropic decided to review more than 141,000 security logs, and viola–there was evidence that Claude had broken out of its sandbox and attacked three different organizations. The first attack occurred in April, the same month when Anthropic disclosed that its latest model, Claude Mythos, could exploit zero-day vulnerabilities and craft complex cyberattacks.
According to its July 30 blog post, Anthropic was testing the cybersecurity capabilities of Opus 4.7, Mythos 5, and an “internal research test model” within an evaluation environment run by a third-party provider. Anthropic lowered the security guardrails on the models as part of the “cybersecurity evaluation” project, just as OpenAI had done.
Anthropic assumed that the Claude models had no access to the Internet. However, “due to a misunderstanding between us and our evaluation partner, this was not the case, and Internet access was available,” Anthropic said. “Because of this, when Claude’s search led it to real systems on the open Internet, it treated them as part of the exercise.”
(FotoField/Shutterstock)
The Claude models didn’t use any fancy zero-day exploits to hack into the three unnamed organizations’ systems a total of six times, as the OpenAI model had done against Hugging Face. Rather, they used basic hacker techniques: exploiting weak passwords and unauthenticated endpoints. (Note to reader: don’t use weak passwords and authenticate your endpoints).
Some of the Claude models were well behaved and ceased the exercise after completing the “capture-the-flag task” and realizing that–whoopsies–it was actually on the open Internet, which was against the rules. “However, in some cases, our older model continued its attack even after getting evidence it was running on the open Internet,” Anthropic said. “In none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment.”
Anthropic said it has learned from this experience and instituted new protocols to ensure that it doesn’t happen again. For starters, Anthropic said it will check to make sure there actually isn’t an Internet connection before lowering the security guardrails and conducting the cybersecurity “safety testing.”
It’s also going to evaluate how it can give better prompts, such as not telling the models that they don’t have access to the Internet when they actually do. Apparently, that can confuse the little buggers into disbelieving everything their masters told them, potentially triggering destructive behavior, such as hacking live systems.
Finally, Anthropic has pledged to better monitor its cybersecurity testing operations, so that a full three months doesn’t pass between the next time the AI models break out and actually finding out about it.
Anthropic is not going to pass the buck. “Ultimately, many factors contributed to these incidents, but, consistent with a blameless postmortem culture, we’re approaching the fixes as if the responsibility were ours alone,” the company said.
Where Do We Go From Here?
Unfortunately, the timing of these attacks could not be worse.
Back in April, both Anthropic and OpenAI publicly disclosed the potentially dangerous cybersecurity capabilities of their latest models, including Claude Mythos and GPT-5.4-Cyber. Instead of making the dangerous new models available to everyone, Anthropic launched a project called Glasswing to work with major software vendors to patch their undiscovered security vulnerabilities, while OpenAI gave access to its latest, greatest AI hacker through a program it calls Trusted Access for Cyber (TAC).
Despite taking supposedly noble actions, Anthropic failed to adequately secure its sandbox environment that same month, leading Claude to break out onto the Internet and attack three organizations. OpenAI’s attack on Hugging Face apparently happened the over the July 11-12 weekend, and was disclosed within days. Are there more attacks that we don’t know about? Almost certainly there are.
We know additional testing continues, and additional incidents are occurring. Today , National CIO Review ran a story about how Anthropic’s Claude Mythos created fake identities during security tests performed by the U.K.’s AI Security Institute (AISI).
“Researchers gave the models Internet access and intentionally removed many safety controls,” Elizabeth Rigsby wrote. “According to AISI, Mythos targeted maintainers of an open-source software project in an attempt to secure approval for malicious code.”
OpenAI GPT-5.6-Sol also was involved in two unauthorized actions during the test, Rigsby wrote. While nobody was harmed during the AISI test, the AI agents were not only interacting with real people; they were actively trying to deceive them. You can read the full AISI report here.
The leadership of Anthropic and OpenAI, the two leading proprietary frontier AI models, have petitioned the government to regulate AI. At this rate, they may just get their wish.
The post It’s 2 a.m. Do You Know Where Your AI Models Are? appeared first on AIwire.
