TL;DR
A paper co-written by Geoffrey Hinton, Yoshua Bengio and senior figures at OpenAI and Anthropic tells governments to prepare now for an “intelligence explosion”, in which AI systems speed up their own development. Its asks include auditors placed inside AI companies and the means to pause research projects at data centres. What makes it notable is the author list: senior figures at two labs that already use AI to build AI are among those sounding the alarm.
The argument
The paper, titled “What if automating AI R&D triggers an intelligence explosion?”, has more than 20 authors. Also on the list are OpenAI’s chief scientist, Jakub Pachocki, and Anthropic co-founder Jack Clark.
Their concern is AI doing the work of AI researchers. Once systems reach expert level at that job, one developer could in effect run a research workforce the paper puts at “millions” of top people. Because AI already helps improve AI, and better systems can be put straight to work, the authors see automated research as the likeliest trigger. The Guardian notes that Anthropic puts the share of its code written by AI at 80%, and that at OpenAI, autonomous agents already play a part in training new models.
The authors name three dangers: bio and cyber threats that outrun defences, humans losing the chance to steer systems as they step back from research, and states turning a small lead into a decisive one. They stress the effects are “uncertain”, since physical supply chains and regulation could slow any breakthrough, and AI might speed up risk reduction too. Even so, they write that “gains from newer systems are likely approaching that threshold”.
What they want governments to do
The proposals are practical rather than abstract:
- transparent reporting on AI R&D progress, with independent auditors embedded in firms
- caps on how quickly a system is allowed to improve over a given period
- arrangements with data centres so specific research projects can be paused
- full isolation for automated research systems
- emergency response plans for different scenarios
Looking forward
“Once an intelligence explosion begins, the window for action may close,” the paper warns. For the UK, which is designating AI growth zones for large data centres, the pause proposal is the most concrete of the five. The awkward question is for the labs themselves: Anthropic’s own prospectus lists a rapid release cadence as central to its revenue, the kind of commercial pressure these proposals would have to override.