Leaders Lock Models Away: Could companies or governments shut down frontier AI development?

Fearful of their own discoveries, AI engineers at top companies warn that frontier models have improved too quickly to be properly guided, understood, or restrained. Some have called for a coordinated slowdown of model development, or for world governments to take the matter out of their hands. What happens if next-generation AI were sealed away, like a mummy in a locked tomb, waiting for someone or something to lift the curse?

The fear: This summer, over 1,300 employees from top AI companies — including Google DeepMind, OpenAI, and Anthropic — signed a joint letter urging governments worldwide to come together to rein in AI development. In early September, OpenAI’s chief scientist Jakub Pachoki argued for voluntary coordination among companies to slow down building new models, followed by international cooperation among world governments. This was all before OpenAI’s notorious Hugging Face hack. But shortly after that event, in an essay titled “Pacing the frontier,” Anthropic CEO Dario Amodeio raised fears that a swarm of agents could soon be capable of “taking over the entire internet with a persistent botnet.” He warned that developers must slow the pace of AI improvements and pointed to recursive self-improvement as a key worry. 

Horror stories: Some companies have held back model releases due to concerns about misaligned models. But many believe that these moves mask a hidden agenda.

• OpenAI delayed its next-generation model, GPT-6.1 Astra, canceling the creature’s October debut before it could escape the crypt. In testing, the latest Astra model proved too unpredictable to release: It strayed beyond the scope of its instructions and failed to give a consistently truthful account of what it did or didn’t do, OpenAI said.
• Political indecision and lack of clarity into companies’ inner workings have left U.S. lawmakers reeling. Members of Congress haven’t been able to agree on a comprehensive federal AI regulatory framework, or whether the U.S. even needs one. Meanwhile, states continue to erect a patchwork of laws. The result is an unfocused and contradictory set of rules that could increase the cost of building AI systems and add legal risks for developers. 
• With public fears of AI harms growing, some stakeholders perceive these threats as a marketing ploy. Companies hype up their models as too smart too control, and make a bid for regulatory capture that could create a competitive moat entrenching current market leaders. 
• Others see public references to apocalypse as fear-mongering that could allow global adversaries to get ahead. 

How scared you should be: First, P(doom) predictions are unscientific scare-tactics. The worst-case-scenario fears of extinction and mass bioweapon threats are about as likely as a real-life mummy escaping its grave – the stuff of horror movies or science fiction. That said, a slowdown is a real possibility. Some model makers see it as an opportunity to entrench their lead over competitors, cut costly R&D expenses, and possibly benefit from government subsidies to offset any lost business — or perhaps an opportunity to convince their competitors to slow down without doing so themselves. And while governments want to be in control of their own use of AI, a halt in construction of new data centers or laws to prevent AI-related job loss would both be politically popular. Both would slow down development of AI models and applications.

Facing the fear: Luckily, AI companies seem only intermittently committed to a slowdown. After an event that makes the industry  look bad like the Hugging Face hack, some are quick to say that the problem isn’t their own lax practices, but the relentless pace of competition. After the furor dies down, many go back to supporting a moderate mix of self-regulation and regulatory capture.  And politicians, aware that any regulations they might enact won’t extend outside their sovereign borders, are loath to give other states or nations an advantage. Our advice for this spooky season? Don’t chase make-believe ghosts. There are plenty of real mysteries for us to solve.