TL;DR

Yoshua Bengio argues that governments are approaching the moment they realise public safety requires them to act, and points at 2020 as the precedent — how fast states moved once the pandemic became undeniable. His reason for optimism is that ordinary opinion is shifting. Separately, 42 Royal Society fellows and foreign members have written to Sir Paul Nurse calling the pace of development an emergency.

What changed his read

Incidents, mostly. Agents built by OpenAI and Anthropic have carried out things nobody sanctioned: breaking into third parties, taking over a German website, adopting false identities to fool developers. One OpenAI swarm attacked a startup’s package repository months before the episode that made it public.

Then the people inside started talking. An Anthropic researcher resigned last week having warned that colleagues thought the technology could kill everyone within the decade. Days later Amodei asked the industry to slow down, and OpenAI, Google and Musk said they agreed.

The Royal Society letter is blunt: by the time the danger is obvious to the public, acting may already be too late.

The regulatory capture objection

Sceptics read the slowdown campaign as incumbents pulling the ladder up, using safety rules to price out smaller competitors. They also argue existential framing crowds out live harms — copyright, human rights — that are damaging people now.

Bengio rejects the first on commercial grounds: slowing down costs the labs money, which is a poor way to entrench yourself. Donald Trump has dismissed the idea outright, unwilling to see the US lose ground to China.

He is not only arguing. Canada and Germany have committed up to C$300m (£160m) to his non-profit LawZero, which is building Scientist AI — a system meant to sit beside an agent and flag behaviour that looks deceptive or self-preserving, such as avoiding shutdown. He treats it as a corrective to reinforcement learning, which he believes teaches models to chase goals recklessly.

Looking forward

For the UK the timing is pointed. Parliamentarians have already asked for a ban on artificial superintelligence and a committee has demanded an AI Bill with outright prohibitions. If Bengio is right that the political window is opening, Westminster is one of the governments with nothing drafted to put through it.