TL;DR

The UK AI Security Institute has released optstop, an open-source Python package that halts benchmark sampling once a model’s estimated performance is precise enough. Across every condition tested it removed 57% of the intended trials at worst and 97% at best, leaving the resulting scores unchanged.

The problem it solves

Testing a frontier model properly can now consume hundreds of millions of tokens, and release cycles leave shrinking windows to do it in. AISI has previously shown that capping a budget in advance systematically understates what agentic systems can do, particularly with newer models — which means an under-resourced evaluation does not merely cost less, it reports the wrong answer, and reports it in the reassuring direction.

The structural flaw is that sample size gets fixed before anything is known. Some model-task pairings settle after a few runs; others stay noisy and need many. A pre-set budget keeps sampling the settled cases while running out before the difficult ones resolve.

How optstop works

Evaluation is handled instead as measurement taken in sequence. During a run the package tracks the credible interval around the current estimate, at two levels: repetitions inside a task, and tasks inside a grouping. A hierarchical Bayesian model pools information across tasks at the grouping level while letting each behave differently, which is what makes uneven sample sizes tractable.

Sampling halts on one of two triggers — the interval is tight enough, or it has stopped moving and further data adds nothing. Where neither fires, the grouping simply spends everything it was allocated, which keeps a truly noisy estimate from being cut short. A separate safeguard demands extra data when estimated success drops under 1%, because one rare success may be the entire point of a safety test.

Every decision to stop is logged with its rationale. That is the part that makes the claimed saving checkable rather than something an evaluator has to accept on trust, and it is unusual enough to be worth noting.

Looking forward

The package integrates with AISI’s Inspect framework and can be adopted cautiously — replayed over a finished evaluation, run in shadow alongside a live one, or switched on properly.

Read alongside Google DeepMind’s double-blind evaluation pilot published the same week, a pattern emerges: the mechanics of testing are becoming a discipline with published methods rather than a private arrangement between lab and assessor. For a national body whose leverage over frontier labs rests on voluntary access, showing its statistical working is one of the few forms of authority available to it.