The AI extinction debate is really a debate about institutional power, responsibility and our ability to remain in control.

For years, artificial intelligence has lived between two competing narratives.

One promised abundance: scientific discovery, higher productivity, personalised medicine, better education and machines capable of extending human intelligence.

The other warned of catastrophe: autonomous systems escaping human control, superintelligence pursuing objectives incompatible with ours, perhaps even human extinction.

Until recently, it was relatively easy to classify the second narrative as the territory of AI safety researchers, philosophers and so-called “doomers”.

That distinction is becoming harder to maintain.

Dario Amodei has now argued that the frontier should be paced so that safety mechanisms can catch up with capabilities. Sam Altman has identified two particularly dangerous possibilities: losing control of the future to AI, and allowing AI to concentrate extraordinary power in the hands of a small number of humans. Elon Musk has also backed calls for slowing the race, despite his scepticism towards some of the more dramatic extinction claims. Dario Amodei

When competitors building frontier AI begin converging, however imperfectly, around the idea that the race itself may require constraints, we should pay attention.

But perhaps not for the reason dominating the headlines.

The wrong question

“Will AI kill us all?” is almost irresistible.

It is also a remarkably poor way of structuring the problem.

It collapses radically different phenomena into a single binary proposition.

Either AI destroys humanity or it does not.

But between those two outcomes lies almost the entire territory that actually matters.

An AI system does not have to exterminate humanity to transform human agency.

It does not have to become conscious to exercise power.

It does not have to become evil to produce catastrophic consequences.

And it certainly does not have to become superintelligent before institutions become dependent upon systems they cannot adequately understand.

The important distinction is therefore not simply between safe AI and extinction.

It is between different forms of loss of control.

There is more than one way to lose control

The first is obvious: humans using AI against other humans.

Cyberattacks, fraud, manipulation, autonomous weapons, surveillance and industrialised disinformation belong here. The technology remains under human direction. The problem is what humans decide to do with it.

The second is more subtle: institutions becoming structurally dependent on AI.

Imagine organisations in which decisions increasingly depend upon models that few employees understand; expertise gradually disappears because the machine normally provides the answer; responsibility becomes distributed between vendors, algorithms, managers and operators; and challenging an automated recommendation becomes organisationally expensive.

Nothing has “escaped”.

No machine has taken power.

Yet something important has already been lost.

The institution’s capacity to know why it is doing what it is doing.

The third level appears as AI systems become increasingly agentic.

The problem here is not necessarily intelligence. It is speed, autonomy and interconnectedness.

A human can theoretically remain “in the loop” while becoming practically incapable of meaningful intervention.

An approval button is not control if the person pressing it lacks the information, time or authority necessary to say no.

And only after these layers do we reach the fourth possibility: genuine loss of control over systems whose capabilities and objectives could become incompatible with human interests.

That possibility deserves serious scientific investigation.

But concentrating exclusively on it can paradoxically distract us from forms of loss of control that are already beginning.

The strangest part of the debate

There is another uncomfortable question.

The companies warning us about these risks are also the companies racing to build the systems.

This does not mean their warnings are insincere.

Nor does it mean they should be ignored.

It means something institutionally much more interesting.

We have created a technological race in which the participants may rationally believe that slowing down is desirable while individually having powerful incentives not to slow down.

That is not primarily an AI problem.

It is a coordination problem.

And humanity has encountered versions of it before: nuclear weapons, financial markets, climate change, biotechnology and global commons.

The rational behaviour of each individual actor can produce an irrational collective outcome.

This is precisely why “trust the responsible companies” cannot be a sufficient governance architecture.

A company cannot simultaneously be competitor, rule-maker, risk assessor and ultimate guarantor of the public interest.

External evaluation, transparency and regulation therefore matter not because technology companies are necessarily irresponsible, but because responsibility itself must be institutionalised.

Perhaps the real alignment problem is ours

Much of AI safety asks how we align increasingly capable machines with human values.

It is an essential question.

But there is another alignment problem receiving much less attention:

How do we align institutions, markets and governments with humanity’s long-term interests when technological competition rewards speed?

Suppose tomorrow we solved technical AI alignment perfectly.

We would still have questions about who controls the systems.

• Who decides their objectives.
• Who benefits economically.
• Who accepts the risks.
• Who has the authority to stop deployment.
• And who remains accountable when decisions are distributed across humans and machines.

Sam Altman’s concern about concentration of power is therefore at least as interesting as the science-fiction-like possibility of machines taking power. Axios

Because before asking whether AI might one day become too powerful for humanity, we should ask whether AI could make some humans and institutions too powerful relative to everyone else.

That is not hypothetical superintelligence.

That is political economy.

From fear to institutional intelligence

I do not know whether AI will ever pose an existential threat to humanity.

Neither, despite the increasingly confident percentages circulating online, does anyone else.

Probability estimates for unprecedented events should be treated with considerable humility.

But uncertainty is not an argument for doing nothing.

A low-probability event with catastrophic consequences deserves attention. Equally, high-probability institutional transformations with less spectacular consequences deserve attention too.

The challenge is therefore not to choose between optimism and apocalypse.

It is to build institutions capable of operating responsibly under uncertainty.

Institutions that preserve human expertise rather than quietly replacing it.

Institutions that know when automation should stop.

Institutions that maintain meaningful human authority.

Institutions capable of auditing systems they depend upon.

Institutions in which someone still possesses the knowledge, legitimacy and courage to say:

No.

Perhaps that is ultimately what AI safety means.

Not guaranteeing that machines will never become dangerous.

But ensuring that, as their intelligence increases, ours – collectively, institutionally and politically – does not diminish.