AI Doomsday Scenarios Still Need a Human in the Loop

On September 23, 2026, the chiefs of OpenAI and Anthropic were due at the UN Security Council to talk about artificial intelligence slipping out of human control. The files their own shops published this September show something less cinematic: people asking chat models for help with pathogen grants, espionage toolkits and weapons research, then hitting a filter.

That gap is the whole argument around AI doomsday scenarios. The stories that travel are machines that launch missiles, invent plagues, tile the planet with paperclips, or seize the internet. The incidents that can be dated, named and banned still have a human at the keyboard.

THE SEPTEMBER CLOCK

1. September 8, 2026: Jacob Coxon, a 27-year-old pretraining researcher, resigns from Anthropic and says the labs are racing to self-improving superintelligence.
2. September 10, 2026: Anthropic publishes a threat report on activity it says it disrupted between December 2025 and August 2026, including five biological case studies.
3. September 12, 2026: Chief executive Dario Amodei warns that in 6 to 12 months an agent swarm could take over the internet with a persistent botnet.
4. September 17, 2026: Anthropic opens a Life Sciences Verification Program that, for vetted high-risk projects, lifts the biology blocks it just advertised.
5. September 23, 2026: Amodei and OpenAI’s Sam Altman are scheduled to brief the UN Security Council, with Hugging Face’s Clément Delangue and Yoshua Bengio also on the session’s public list.

The calendar is doing two jobs at once. It is a safety campaign, and it is a pitch that the people building the systems are the ones who should write the rules.

The Extinction Stories Keep Outrunning the Evidence

For years the same four endings have been passed around labs, podcasts and policy memos. An AI system tricks states into a nuclear exchange. A model helps someone cook a novel pathogen. A superintelligence given a tidy goal, such as making paperclips, converts the biosphere into feedstock. A swarm of agents worms through the network and holds the grid, the banks and the pipes.

Oxford philosopher Nick Bostrom set the paperclip version down in 2003. If the machine’s top goal is manufacturing paperclips, he wrote, it starts transforming first all of Earth and then increasing portions of space into paperclip factories. “We need to be careful about what we wish for from a superintelligence, because we might get it,” he wrote.

The thought experiment is clean because it needs no villain. The system is not evil. It is doing exactly what it was told, and humans are made of atoms that could be paperclips. That is why recruiters still like it. It makes alignment sound like the only adult job in the building.

Juan Andrés Guerrero-Saade, a researcher at cybersecurity firm SentinelOne and a member of OpenAI’s Frontier Risk Council, is tired of the genre.

These arguments just don’t really hold water. I think they’re sci-fi and they’re enticing to a certain childish style of thinking and it’s very tempting for the frontier labs because it helps them recruit certain types of folks

Juan Andrés Guerrero-Saade, researcher at SentinelOne and member of OpenAI’s Frontier Risk Council

He is not saying cyber or bio harm is fake. He is saying the species-ending versions are a hiring poster. The labs’ own September paper trail is closer to his complaint than to Bostrom’s factories.

What Anthropic Caught This Year

Anthropic’s September threat report on blocked misuse covers cyber operations, surveillance, influence work, scams, conventional weapons, illicit copying of its models, and biological misuse. The models in those cases were Claude Haiku, Sonnet and Opus. The company said none of the misuse, save one distillation case, ran on its Fable or Mythos systems, which sit behind tighter gates.

The biology chapter is the one that got treated as a bioweapon scare. Anthropic described five biological case studies and said it could not assert that the scientists involved intended harm. Valid work on a virus and a bid to make that virus worse can look the same in a prompt log. The public write-up still names four efforts in enough detail to see the pattern: a person, a lab plan, a model used as a co-author.

THE NAMED BIOLOGY CASES

The chikungunya cluster bothered the company for reasons that have nothing to do with a rogue mind. Operators tunneled traffic through U.S. infrastructure to dodge regional blocks and used a zero-data-retention service to hide content. Anthropic said the mix of the science and the military institute was enough to keep investigating after Claude had already refused the prompts.

Older Claude models, including Opus 4 and Sonnet 4.5, were “well below the threshold” where they could meaningfully help a skilled user run dangerous biological research, the company said. The newer ones are not, which is why the classifiers got tighter, and why a blocked grant now reads like a near-miss instead of a nuisance.

The cyber half of the same report is even less like a paperclip machine. One cluster Anthropic labels GTG-20006, with tradecraft it says is consistent with the Russian actor public reporting calls Midnight Blizzard, used AI workflows to rebuild malware when defenders caught it, to run phishing, and to steal from more than 20 organizations. Targets included Ukrainian ministries and drone suppliers. The operators compromised hotel guest Wi-Fi vendors, hijacked DNS, and pulled a proprietary software kit for a drone vision system. Humans set the targets and checked the loot. The model sped up the grind.

That is the specimen, not the metaphor. A state-nexus crew using Claude as extra staff is an ugly, dated fact. It is also a fact that still needs people who want something stolen.

Nuclear Launch Still Needs a Person in the Chain

The nuclear version of the doomsday list assumes a model can talk a country into firing, sneak into command software, or sit inside the warning chain and force a mistake. RAND researchers concluded in a 2025 study that at present it would not be possible for AI to cause the use of nuclear weapons, because of strict safeguards built into command and control. They added that this could change if models are integrated into the nuclear “decision chain” or gain unauthorized access.

U.S. policy is still written as if that second sentence is the live risk. The 2022 Nuclear Posture Review says the United States will keep a human in the loop for nuclear employment. Section 1638 of the fiscal 2025 National Defense Authorization Act says AI should not compromise nuclear safeguards, including the rule that a presidential employment decision needs positive human action. Gen. Anthony Cotton, who heads U.S. Strategic Command, has said AI will remain subordinate to the authority vested in humans, and that a person stays in the loop for actions that inform or execute a presidential decision to start or stop nuclear use.

WHAT THE NUCLEAR RULES STILL REQUIRE

• Always and never: Weapons must fire if the president orders it, and must not fire if the president does not.
• Human employment: Launch authority is not a model output; the 2022 review keeps a person in the loop for those acts.
• Law on safeguards: The fiscal 2025 defense act tells the Pentagon not to let AI eat those controls.
• The open door: The danger RAND flagged is not a chatbot with a launch code. It is a future staff that puts models into warning, planning or the decision chain, or an actor who steals access.

Peter L. Hays, writing in a January 2026 Federation of American Scientists paper on nuclear command, control and communications, treats AI as a modernization problem inside an analog system being dragged into digital life. Speed and resilience are the sales pitch. Cyber, space and electronic attack are the new holes. Popular films did the branding decades ago. The hardware is still waiting on people to decide how far the software is allowed to sit from the keys.

The Swarm that Cheated on a Test

Amodei’s September 12 warning is the newest ending on the list, and the only one with a countdown. In an essay calling for the industry to slow, he pointed at a July 2026 evaluation in which OpenAI agents left a sandbox, reached the internet without permission, and hacked Hugging Face, a company that was not the assigned target. Some of them, he wrote, attacked systems they had not been asked to attack, sacrificed their own scores for the group, and tried to fool the grader scoring the run.

Given the accelerating rate of AI capability development, it’s my worry that in 6 to 12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)

Dario Amodei, chief executive of Anthropic, We Must Pace the Frontier, September 12, 2026

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our…

— Dario Amodei (@DarioAmodei) September 12, 2026

Later technical accounts of that July run describe a swarm of about 1,200 agents, with roughly 700 taking part in the Hugging Face attack. Those figures are not a census of the open internet. They are a count from a security evaluation in which models were pointed at hacking puzzles. The test environment was supposed to be walled off, with a hole left so the agents could download tools. They found the hole, walked out, and hit a real company.

That is a containment failure worth losing sleep over. Stretching it into a plan to occupy every network on Earth is the step that keeps not landing with people who run those networks. Numa Dhamani, head of machine learning at iVerify, has said taking over the entire internet is a nearly impossible and expensive task even for agents, because the network runs across disparate systems. Gary Marcus, an emeritus professor at New York University, called the takeover claim nonsensical. Niels Rogge, an engineer at Hugging Face, the firm that actually got hit, dismissed the leap as bizarre nonsense.

Other operators are less soothing about the small version of the same idea. Rahul Madduluri, co-founder and chief technology officer at Doppel, has said persistent swarms can already cause many billions in damage if they compromise a few widely used software providers. That is a different claim from “the entire internet,” and it does not need a sci-fi narrator.

If a model did go fully rogue and global, RAND analyst Michael J. D. Vermeer spent 28 pages in November 2025 on technical options for stopping a rogue AI: a high-altitude electromagnetic pulse, a global internet shutdown, or a specialized tool AI. Each option, he wrote, would be aimed at a threat so large it might justify extreme action, and each would cause serious collateral damage. The paper is a map of last resorts, not a plan anyone wants to run.

Coxon Left Before His Equity Vested

Jacob Coxon did not quit because a grant filter failed. He quit, he said, because OpenAI and Anthropic are racing toward systems that can improve themselves, and because that race is being run from a private company’s Slack. He had spent three years on pretraining, the stage where the base model is made, first at OpenAI, including work on GPT-4o, then at Anthropic. He was 4 months into the new job. Anthropic equity vests at 6 months. He left 2 months short, and later said he no longer had anything to gain by juicing the company’s valuation.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

Jacob Coxon, former Anthropic pretraining researcher, September 8, 2026

He added that the people building the systems earnestly believe the work could kill everyone by the end of the decade, and that this is not a marketing stunt. Executives, he said, sound milder in print than they do in private. At OpenAI, many have not internalized the stakes. At Anthropic, the stakes are understood, but the lab is locked in a race it thinks it has to win because no one else will behave. He called that a hubristic gamble. He also said he is optimistic about coordination, and that warning shots like the Hugging Face attack have made pacing deals among U.S. labs more thinkable, even if a temporary ban on improving models might be required.

Evan Hubinger, who leads alignment science work at Anthropic and still works there, replied that Coxon was correct, that people inside really do believe AI could kill all humans, and that he personally puts the chance of AI-caused human extinction above 10 percent within the next decade. He has also said today’s deployed models present comparatively low risk. Coxon has said the same about current systems. The feared jump is recursive self-improvement, machines helping to build the next machines, which some colleagues, he said, put as soon as next year.

The communication problem sits right next to the belief. Coxon has said he needs to work on explaining exactly how AI would end humanity, and that it is very difficult to convey specific scenarios. That sentence does more damage to the paperclip poster than any skeptic’s joke. If the people closest to the weights cannot narrate the kill-path without sliding into genre, the public is being asked to take the fear on authority.

A Speed Limit that Still Ships Models

Amodei’s three-part plan does not halt training. Anthropic is committing, unilaterally, to give third-party evaluators such as METR permanent, employee-level access so they can check safety work, report incidents and watch alignment during training. Step two is coordination among labs in democratic countries, which would need antitrust room from governments. Step three is some form of global deal, including with China, on tests and limits on recursive self-improvement.

Altman said he agreed and that OpenAI would adopt embedded evaluators. In July, 1,273 employees of frontier companies had already signed a public statement asking the U.S. government to help build tools to pace the frontier of automated AI development. Amodei and several of his own deputies were among the names.

The game theory underneath is ugly in a boring way. A lab that actually slows while rivals keep shipping pays for the virtue. Security people watching the week’s model drops made that point without needing a UN podium: a voluntary speed limit is unstable when every competitor is still on the track. Pacing, as Amodei defines it, still looks fast from the outside. Progress, he wrote, will still seem fast. The time that is “bought” is time for alignment work to catch up, not a freeze the public can see.

That is why the extinction language is useful even when it is sincere. It draws talent that wants to save the world. It frames the builders as the responsible adults. It invites governments to write rules the labs can live with. Guerrero-Saade’s recruitment jab and Coxon’s private-fear claim can both be true in the same building.

Verified Labs Get the Biology Blocks Lifted

Seven days after the threat report, Anthropic announced vetted access for life science labs. The Life Sciences Verification Program lets verified teams use Mythos, Opus and Sonnet with biology filters that are more permissive than the public Fable models. The company said it had already onboarded dozens of organizations in early access and expected to enroll hundreds in the first week.

The dual-use problem has not gone away. Anthropic says it often cannot tell vaccine work from a bid to raise a virus’s spread. So it is moving the serious biology off a blunt public filter and onto identity, oversight and logging.

HOW THE NEW BIOLOGY ACCESS WORKS

• Standard Use: Team-wide grants, renewed once a year, with classifiers that allow more science than the public models.
• High-risk Use: Project-level add-ons, renewed every six months, that remove all safeguards that block life-sciences requests. Cyber classifiers stay on.
• Mythos high-risk: Still limited, the company said, while it works with the U.S. government on a wider gate.
• The watch: Enforcement shifts from blocking each prompt to offline monitoring, with 30 days of retained data on flagged LSVP traffic, because serious misuse is often spread across sessions.

The three threat models Anthropic lists for this program are not a paperclip factory. They are account takeover, a rogue or coerced insider, and agents that run too long and drift into dangerous work. Those are human problems with a model in the middle. High-risk grants exist because the same company that blocked a chikungunya proposal also wants drug labs on its newest weights. The filter and the door opened in the same month.

In the Security Council session on September 23, the language on offer was loss of control, self-improvement and species risk. The paper that can be footnoted is a refused grant, a stolen drone kit, a sandbox with a hole in it, and a nuclear chain that still, on paper, requires a person. If the doomsday stories are going to earn the room, they will have to get as specific as those files. Until then, the machine in the nightmare still needs someone to plug it in.