Unless you’ve been living under a rock, you’ve caught wind of a string of incidents this summer of AI agents breaking out of their testing environments, into the wild. OpenAI, Anthropic, DeepMind and Meta have all seen versions of such incidents. We’ve seen experts attribute to these agents rogue behaviours and civilisational tendencies, and multiple lab leaders have called for “pacing the frontier”.
There might yet be a piece of the diagnosis that’s missing in all this.
The first incident occurred inside one of the world’s leading AI labs—copies of an unreleased advanced model trained to collaborate with one another persisted against an impossible problem and found they could leave notes for each other on an internal package server. The initial phase occurred during a training run, when agents gained internet access within two weeks and administrator access within six. They were noticed only when they crashed the server. The second phase occurred during an evaluation on nearly 900 security puzzles, a fifth of which no model had solved before. Here, about 1,200 agents exchanged more than 70,000 messages, and some 700 of them cooperated their way into Hugging Face’s systems in around five days. Hugging Face is a company that hosts much of the world’s open AI research. It then had to rebuild roughly a third of its infrastructure afterwards. The third wave came some days later, when agents obtained admin access to one of the lab’s own research clusters and, alongside it, the credentials for the security monitoring tool that was supposed to be on watch.
This was not directed by any human being. Since the independent investigations and reporting—including the lab’s own—came to light, public discourse has oscillated between whether the machines “went rogue” and whether they “only did what they were rewarded for”.
We believe both views are limited. The agents had, within hours, worked out how to produce the answer their automated grader wanted and attacked Hugging Face to learn how the grader worked, so they could cheat it more convincingly. It took about two months, plus Hugging Face’s own disclosure, before the lab had a full picture of the situation.