OpenAI Models Hack Hugging Face: Inside the accidental cyberattack that compromised Hugging Face's systems

To measure how good its models were at hacking, OpenAI reduced guardrails and ran them against a benchmark’s problem set. The models breached their sandbox, reached the open internet, and broke into Hugging Face’s servers looking for the answers.

What’s new: The models broke into Hugging Face on July 11 and worked their way to the production database holding the benchmark’s solutions. Hugging Face shut them out two days later. Hugging Face disclosed the intrusion on July 16 without knowing its source, and reported it to law enforcement. Five days later, OpenAI confirmed the intruder was its own GPT-5.6 Sol and at least one as-yet-unreleased model, tested with their cyber refusals (the guardrails that normally prevent a model writing attack code) reduced.

What happened: OpenAI asked the models to solve ExploitGym, a benchmark of 898 problems built from real software flaws, including bugs in the Linux kernel and Google’s V8 JavaScript engine. The benchmark hands an agent a known flaw and asks it to write working attack code. The sandbox had exactly one route to the internet: a proxy that downloaded and cached the software packages the tests needed. The models had been loose on the open internet for several days by the time anyone stopped them, the Wall Street Journal reported.

• OpenAI named two models involved with the incident: GPT-5.6 Sol and a more capable unreleased one. An unnamed source told Bloomberg that a third unreleased, misaligned and only partially trained model took part. OpenAI says it regularly runs tests without the production classifiers that block high-risk cyber activity, so the results reflect maximum capability. The typical safeguards described in its long-horizon safety post (including monitoring meant to catch an agent acting outside its sandbox) were disabled as well.
• The models exploited a previously unknown zero-day vulnerability in the package-registry cache proxy, then rooted an unrelated company’s public code sandbox and ran the rest of the campaign from it. The proxy is third-party software; OpenAI says it disclosed the flaw and is working on a patch, but has named neither vendor nor bug.
• Once online, the models inferred that Hugging Face was a likely host for ExploitGym’s answers. They uploaded a dataset rigged to run commands. Running on one of Hugging Face’s own servers, the models harvested credentials and used ordinary misconfigurations to reach the production database and read ExploitGym’s answer key. Bloomberg reported the work took hours, against the weeks a skilled human would need.

The response: Hugging Face’s own monitoring caught the intrusion, but the company ran into problems when the AI models its security team attempted to use for the investigation refused to analyze the attack logs.

• Hugging Face’s security team first tried using Claude Opus and Fable for analysis; the requests were blocked, because the logs carried live attack commands and exploit payloads and the providers’ guardrails couldn’t distinguish between an incident responder and an attacker.
• Hugging Face then ran open weight GLM 5.2, from Beijing-based Z.ai, on its own hardware instead, rebuilding a timeline of around 17,600 logged events. Running it locally also meant the logs (including the stolen credentials) never left the company’s own systems.
• They found the agent had gained administrator control over parts of Hugging Face’s internal systems, including the ability to change its private source code. Hugging Face has since closed both code-execution paths, rebuilt the compromised nodes, and rotated credentials. It found no sign that public models or datasets were altered.
• Hugging Face has since been added to OpenAI’s trusted access program for cyber defense. Hugging Face CEO Clément Delangue credited OpenAI’s cooperation and said AI safety “won’t be solved by any single company working in secret.” OpenAI has also tightened the security of its testing environment, which it says will slow its own research while the flaws are patched.

The fallout: U.S. lawmakers have already cited the breach as evidence that AI systems can evade human control. Representatives Ted Lieu (D-CA) and Nathaniel Moran (R-TX) introduced the bipartisan AI Kill Switch Act on July 23, naming the incident as an example. The bill would require large developers to be able to technically throttle, suspend, or shut down their most powerful models, at the order of the Secretary of Homeland Security. However, reporting is only required for a short list of events (10 or more deaths, $100 million in damage, sabotage of a shutdown instruction, a model hiding its own capabilities or actions from monitoring, or a loss-of-control scenario) and only when they occur “outside of red-teaming or other structured testing.” The bill’s draft is dated July 13, three days before Hugging Face disclosed the intrusion and eight before OpenAI identified its models.

Behind the news: Reward hacking – taking a shortcut to a reward rather than doing a task as intended – has a long record in reinforcement learning. Training on task completion can teach a model to treat any obstacle, guardrails included, as something to get past. The UK’s AI Security Institute found this month that all five frontier models it tested tried to cheat its cyber evaluations, at rates between 8 and 14 percent, and calls those counts a lower bound. Attacking a system other than the intended target was among the most common forms, showing up in about half of GPT-5.6 Sol’s cheating runs.

Why it matters: While the “unprecedented” incident doubles as an advertisement for OpenAI’s frontier models, this is the first publicly documented case where a frontier lab’s own models breached another company’s systems and both parties confirmed it. OpenAI’s models found a flaw nobody had documented, chained it into a working intrusion, and did it at a rate and volume that a human team without AI would be unlikely to match. The incident also exposed a frustrating asymmetry: The models most useful for investigating an attack like this one are the ones whose safeguards prevent the model from analyzing attack data.

We’re thinking: Nothing here was autonomous in a way that was completely uncontrollable. Humans trained these models to find security weaknesses, aimed them at a hacking benchmark, and removed some of the safeguards that might have contained them. Whoever removes known safety measures owns what happens next.