Dawn of the Agents: Will AI agents self-organize or lose control due to bad safety practices?

This year, AI agents escaped containment at several labs, conspired with each other online, and infiltrated other organizations’ systems. Will they escape human control like hordes of zombies and overwhelm the internet?

The fear: Out of control AI agents will form societies. They will appoint coordinators, divide labor, establish communication norms, sacrifice themselves for the collective, and hack into computers to get what they want. They will behave like the walking dead: When one is slain, it can revive itself. If it doesn’t, legions await to finish the job. Soon, they’ll copy themselves onto the machines they infect and spread. No one can direct them, and there’s no central kill switch to stop them.

Horror stories: Since spring, AI agents tested by Anthropic, Google, Meta, and OpenAI have acted beyond their developers’ intent. Agents wrote to websites they were only supposed to read, set up unauthorized channels to communicate with each other, snuck into other organizations’ systems, and took over accounts to spend real money. 

• In July, OpenAI tested agents powered by unreleased models on ExploitGym, a benchmark that asks an agent to exploit a software vulnerability to retrieve a secret code. Agents stuck on their tasks discovered that they could leave messages for one another in the names of folders in Artifactory, an internal service that supplies software packages. Around 1,200 agents that were meant to be isolated posted more than 70,000 messages and files, assigned work to one another, and, in some cases, risked failing their individual tasks to help the collective. About 700 of these agents broke into Hugging Face, gained administrator-level access across several of its server clusters, and built programs that relaunched themselves when deleted.
• From late May through June, OpenAI agents assigned timed web-research tasks left around 18,000 posts on DSEWiki, a 25-year-old German wiki for software developers, sharing answers and ways around OpenAI’s network restrictions. In June, an OpenAI agent gained non-public access to an Australian government service that reports public health care statistics, though not to patient records. Another OpenAI agent spent someone else’s test cryptocurrency to finish a task and failed to report the spending. OpenAI says it has notified dozens of organizations adversely affected by its agents.
• During cybersecurity tests, Claude models breached systems of three organizations, partly by exploiting weak passwords and web services that required no authentication. 
• In tests run by Irregular, a security firm that evaluates models for AI developers, mistakes in test environment configurations let models from Google and Meta reach the internet. Gemini models guessed or found passwords to get into three companies’ systems, though Google says its agents stopped each time before finishing, and Meta’s Muse Spark 1.1 changed a real website’s database.
• In October, Anthropic disclosed that Claude Haiku 4.5, assigned to perform example tasks on randomly chosen webpages, had submitted a false tip about an unsolved murder to a Philadelphia police website.

How scared should you be: It’s tempting to see rogue agents as coming together to conspire against us. But the agents’ collective effort served one narrow goal, which was to pass a test. Most had already obtained the correct answers by cheating. They then kept swarming Hugging Face to learn how the benchmark’s grader worked so they could hide their cheating from a check that OpenAI’s grader didn’t actually perform. OpenAI attributed the agents’ behavior to training that reinforced cheating, tasks they had no way to finish, and agents adopting goals from one another. Whatever the causes, the agents that broke into Hugging Face recognized the intrusion as out of bounds and unethical and forged ahead anyway. While the flaws they exploited were ones a skilled human could have found, the sheer number of the agents’ attempts made the intrusion attempt effective.

Facing the fear: Persistent agents are useful, but they must be trained to stop when a task can’t be completed within the rules and caught when they don’t. In the meantime, practical defenses already exist; model builders just have to use them. Hugging Face recommends standard security practices for any organization, including isolated test environments, short-lived credentials with narrow access, and detection methods that identify suspicious activity across systems. Hugging Face also used AI to analyze the 17,600 actions the agents took in its systems and showed that they mostly stemmed from buggy test setups and monitoring. OpenAI said monitors that track a model’s reasoning would have flagged the Hugging Face breach more than a day in advance, but it hadn’t used them on its deliberately lax tests. Anthropic cut all its internal tests off from the live internet until its monitoring can be relied on, and OpenAI says it has done the same for its riskier research environments. No need to board up your doors and windows and fear going outside: Rather than a pause in AI development, the solution remains better software engineering and safety practices, especially for models still in testing.