TL;DR

Off-by-1 Labs, the research arm of 1Password, generated 6,080 security patches for six CVEs disclosed in recent months, using ChatGPT 5.5 and Claude Opus 4.8, then graded every one. Only 26% both closed the flaw and left application behaviour untouched. Nearly half left at least one exploit path open. The team’s conclusion is unambiguous: an AI-written patch that no human reviews carries negative expected value by a wide margin.

Where the other three quarters went

The failure modes matter more than the headline rate. Just over a fifth of patches closed the original flaw but altered application behaviour — the example given is logic silently flipping from an allow list to a deny list, which fixes the CVE and breaks the product. 2.3% fixed the issue while opening a fresh security hole. 2.2% managed both to miss the vulnerability and introduce a new exploit path.

Even among patches graded successful, over a third were judged fragile: code that blocked one specific route in, typically by escaping particular characters, without touching the underlying defect. The team coined an acronym for the output — FLAWED, for fix-like artifacts with embedded defects — and has released an evaluation harness under that name for organisations wanting to grade their own fixes.

Guidance helps, and wrong guidance is fatal

The most useful finding for security teams concerns context. Given correct initial guidance, model success rose to 65%. With none at all it sat at 50.4%. Given incorrect guidance it collapsed to 15.2%. A human developer working through vulnerable code stands a fair chance of spotting that the brief is wrong, the authors argue. The models did not, and were actively derailed by it.

The economic case looks strong in isolation. A successful clean patch cost $6.74 on average, including all the failed attempts behind it. The authors’ warning is that this figure omits the expensive part — reviewing a large volume of near-identical, subtly wrong patches imposes a cognitive load they suspect exceeds the effort of a developer simply fixing the vulnerability themselves with AI assistance and their hands on the controls.

Looking forward

This is a useful counterweight for UK security teams currently being sold autonomous remediation. It also sits alongside the past fortnight’s containment disclosures as evidence pointing the same way: capable models given an objective and insufficient supervision produce confident output that does not survive inspection. The researchers’ phrase for accepting a 1-in-4 process unexamined is cognitive surrender.