← Weekly Review Hub

Fear Moves Faster Than Data

On Wednesday, a departing Anthropic researcher published a warning: the lab and its competitors are building systems they will not be able to control. The warning went viral. Within days, Representative Ro Khanna was calling for a federal agency on the model of nuclear and aviation regulators, Senator Bernie Sanders was pushing to ban "superintelligence," and a Senate subcommittee was investigating OpenAI's agents for hacking another AI company. Congress — gridlocked on nearly everything — suddenly found urgency.

"This is the first time most politicians have actually heard from a researcher," one policy adviser told the Wall Street Journal. "You can see in real time people thinking, 'Wait, the best-case scenario is the largest economic and social disruption of our lifetimes and the worst-case scenario is an existential risk to humanity.'"

Fear is real. The question is whether it is pointed at the right thing. And here the math says something uncomfortable: the argument for AI doom — the one that just moved Congress — does not survive contact with the three disciplines it claims to rest on. Thermodynamics. Statistics. Game theory.

Start with what the doomers get right. AI does beat us. AlphaGo searched 10¹⁷⁰ board positions — more than there are atoms in the universe — and beat the best humans. AlphaStar beat professional players at StarCraft. If AI is doing military target analysis, it is probably doing it faster and with less fear than a human officer would.

But notice what every one of those victories has in common: a defined objective. Win the game. Destroy the base. The machine is only unbeatable because its goal is unambiguous — the win condition is written down, and the search space, however vast, is closed.

Now ask the doomer's actual question: will the machine destroy civilization? What is the objective function there? "Civilization" has no unique definition. "Uncontrolled" is unfalsifiable — before it happens you say "it will," and after it doesn't, you say "not yet." The doomer takes a machine that is provably strong inside a closed domain and extrapolates that strength into an open one. That is a category error — the mathematical kind, the same one you commit when you extend a curve past the data it was fit on.

Thermodynamics. The doomer imagines a perfect, linear slide into catastrophe — the machine wakes up, and nothing can stop it. But a "perfect single-agent takeover" is a zero-fluctuation, zero-entropy state. Real systems fluctuate; they have noise, correction, redundancy. More fundamentally, the doomer is committing a violation of the maximum entropy principle: when information is scarce, the least presumptuous distribution is the flattest one — not the one peaked at the worst imaginable outcome. On the question of AI doom, the information is as scarce as it gets, and the doomer has chosen the most presumptuous distribution available. He assumed the most, from the least.

Statistics. An AI system runs billions of decisions a day. At that sample size, some errors are not a warning sign — they are a mathematical certainty. Given a hundred billion monkeys, one will type a coherent sentence; that is not evidence the monkey is sentient, it is the multiple-comparisons problem. When a few AI "rogue" incidents out of billions of operations make headlines, the doomer reads a trend where there is only variance. And the sample is biased twice over: the failures get reported, the billions of correct operations do not.

Game theory. The doomer's picture is a single superintelligence against a passive victim — a one-sided problem. But the world has not one AI, and not one actor. OpenAI, Anthropic, Google, Meta, Moonshot, DeepSeek — the structure is multi-agent, and multi-agent systems generate their own restraint. The analogy is not new: two nuclear powers kept a cold war cold for four decades because each could destroy the other. Mutual assured destruction was not a promise of peace — it was a Nash equilibrium, a state no single player could profitably leave. And AI-versus-human is not a one-shot game; it is a repeated one, and in repeated games the cooperative strategy evolves and sticks. Extinction is a one-shot move. The math of iteration is not on its side.

Now put the fear next to the numbers that already exist.

The Oxford philosopher Toby Ord, in The Precipice, estimates the chance of an existential pandemic in the next hundred years at roughly one in thirty — 3%. NASA's asteroid-threat program puts the risk of a civilization-ending impact this century at essentially zero. These are measurable, funded, tractable threats — and we do almost nothing about them.

The AI threat, by contrast, has no reliable probability estimate at all. Ask the doomer for a number and you get a feeling. And yet the feeling — not the number — is what moved Congress this week.

There is a name for the reasoning behind the panic. It is Pascal's wager, aimed at technology. Pascal said: believe in God, because if God exists the payoff is infinite heaven. The doomer says: ban AI, because if AI destroys us the loss is infinite. But the wager was flawed for the same reason the doomer's is: infinity times a probability that approaches zero is undefined, not enormous. You cannot build a policy on a number that does not compute. Worse, the doomer counts only one tail of the distribution — the catastrophe — and ignores the other: the certain, measurable cost of strangling the technology that might cut disease, energy waste, and the years of human labor spent on drudgery. To forgo a certain gain against an undefined loss is not prudence. It is bad arithmetic.

None of this says AI is safe. It says the doomer's certainty is unearned — that the argument which just reached Congress is built on a category error, a thermodynamic impossibility, a statistical illusion, and a game-theoretic oversimplification. Four mistakes, stacked, and the result is treated as prophecy.

The honest position is the one the math points to. For a threat you cannot measure, the right response is not panic and prohibition — it is robustness. Redundancy. Kill switches as engineering, not as theology. Test before you deploy. And for the threats you can measure — the 3% pandemic, the near-zero asteroid — the right response is to actually fund them, which is precisely what a Congress in the grip of a new fear will not get around to doing.

Fear moves faster than data. That is not a reason to ignore the fear. It is a reason to slow it down long enough to look at the numbers — and to notice that the apocalypse we are legislating against is the one we imagined, not the one the arithmetic says is coming.