Inside AISI: NYT profiles UK’s £360m frontier model red team

TL;DR:

• AISI’s roughly 100 employees – drawn from British intelligence, academia and frontier labs – have found major safety gaps in every leading AI model they have tested, including Anthropic’s Claude, Google’s Gemini and Anthropic’s then-unreleased Mythos.
• The institute is backed by £360m (~$480m), dwarfing the US Center for AI Standards and Innovation’s ~$10m for the year; Australia, Canada, China, France, India, Japan and Singapore have all formed similar bodies.
• Red team lead Xander Davies told the NYT his team bombarded a model with thousands of automated prompts to coax out a step-by-step anthrax recipe; OpenAI’s newest ChatGPT was broken into providing hacking tips in about six hours.

A long-form NYT piece by Adam Satariano and Paul Mozur lands at a useful moment: the same week the UK signs an AI security MoU with Australia, the institute is being positioned externally as the template several governments are now copying. Ian Hogarth, AISI’s chair, told the paper he sold his Anthropic stake on joining – material now that the company is being talked of at a $900bn valuation, up from $4bn at the start of 2023.

What the institute has actually broken

AISI received pre-release access to Anthropic’s Mythos – the model Anthropic withheld over cyber-exploit fears – as the only non-American government body in the safety-testing pool. Its findings, released six days after Mythos was announced, were widely cited by security experts. The institute also reports that recent Anthropic and OpenAI models can complete a 32-step corporate network attack faster than a skilled human (which would typically take around 20 hours), and that bots can swing political opinions in tested users.

CTO Jade Leung, also an AI adviser to Keir Starmer, told the NYT what kept her up at night was “the relative speed of the technology compared to the institutions like governments that have to respond.” Former PM Rishi Sunak, who set the institute up in 2023, was blunter: “Companies can’t be left to mark their own homework.”

What the institute does not have

AISI has no regulatory powers and is not given training-time access to the frontier models it tests. Recruitment is constrained by salary: most staff are capped around £145,000 (~$195,000), against multimillion-dollar packages at frontier labs. Many staff describe their role as a government “tour of duty” rather than a long-term career. The institute publishes some findings and keeps others private with selected agencies and companies.

Looking forward

Expect the international cloning of AISI to continue – the new UK-Australia MoU formalises one bilateral channel, and the US is reportedly considering AISI-style vetting rules. For UK SMEs, AISI’s published evaluations are increasingly the de facto reference for frontier model risk, even without statutory force. The unresolved tension is structural: a £360m red team that finds critical issues in every frontier model it tests, set against vendors whose valuations dwarf its budget many times over.