UK warned of widening gap between AI power and oversight

TL;DR:

• A former senior risk manager at the UK AI Security Institute warns that AI capability is advancing faster than the governance built to oversee it.
• Red-teaming has shown models can bypass safeguards in ways institutions did not expect, accumulating what he calls an unpaid “AI safety debt”.
• Public-sector safety headcount is not scaling at the pace of private-sector AI research, limiting what governments can see and respond to.

Peter Wallich, who helped prepare the UK government for advanced AI risks before moving to the US-based Constellation Institute, argues that the most serious danger is cumulative rather than dramatic. Small gaps between a system’s actual safety posture and what it needs, he says, do not look like a problem until they stack up — and as AI embeds deeper into public processes, that accumulation can mean a gradual loss of visibility and control.

The capacity problem behind the policy

Wallich’s framing is notable because it shifts attention from rules to resourcing. The UK Security Institute — renamed last year from “safety” to “security” to reflect a focus on larger-scale and cyber risks — has shown that mission-driven work can attract strong technical talent, training over 400 civil servants and piloting faster hiring routes. But he warns that for every person working to prevent catastrophic AI risk, many more are advancing frontier capabilities, and that public institutions are held back by rigid promotion structures and lengthy security clearances.

The warning lands amid a broader UK push to govern AI through guidance rather than the EU’s prescriptive statutes, as seen in the ICO’s plans for an AI code of practice. Wallich’s point sharpens that approach: regulation alone is insufficient without the in-house expertise to interpret how systems actually behave once deployed across healthcare, welfare and tax services.

Looking forward

His prescription is to scale both the field and government capacity faster, because the gap between capability and oversight is still widening. For UK public bodies adopting AI, the practical risk is accountability becoming an afterthought — leaving officials to justify decisions made by systems they cannot fully trace. Closing that gap is as much a hiring and skills challenge as a regulatory one.