TL;DR

Three AI application-security scanners were run over the same codebase and concurred on roughly 5% of what they found. Rerunning a single tool against unchanged code reproduced about 17% of its own previous results. The figures come from Contrast Security, which sells competing runtime tooling, so read them with that interest in mind — but the reproducibility problem they describe is straightforward to verify.

The economics are the interesting part

Scanning two million lines with AI tools cost roughly $315 in API charges. Working through what came back cost around $128,000. That ratio is the finding worth carrying into a budget meeting: the scan is nearly free and the triage is not, so anything that inflates the output volume transfers cost onto the team least able to absorb it.

Contrast’s CISO David Lindner puts the operational objection plainly. The tools contradict one another, and contradict themselves from run to run. That is survivable, he argues, “when AI is one input among several” but becomes “a problem when it becomes the system of record, because that is what decides what my team works on Monday morning”. That distinction is the practical takeaway regardless of who funded the research.

The backlog it lands on

The surrounding numbers describe teams already underwater. A monitored application carries an average of 106 vulnerabilities in first-party code, 22 of them high or critical, against a remediation rate of about 3.4 per application each month. The average critical fix runs to 92 days. Log4Shell and Spring4Shell, both years old, still appear in live telemetry.

Meanwhile the window has closed. Mean time to exploitation exceeded two years in 2018 and fell below one by 2021; most vulnerabilities exploited in 2025 were weaponised inside three weeks. Adversaries touch a typical application every four minutes, and 42 attempts per application each month are confirmed viable rather than mere probing. Crowdsourced defence is also thinning, with HackerOne closing the Internet Bug Bounty to new submissions in March 2026, and Node.js shelving its own scheme once that funding went.

Looking forward

For UK buyers this sharpens a question raised by our recent coverage of insurers rewriting cyber cover and of the hundred-plus firms warning the window to fix defences is closing. Sellers increasingly offer AI triage as assurance. If two tools looking at one codebase share a twentieth of their conclusions, that assurance needs evidence attached — ask any vendor for run-to-run reproducibility figures before their output becomes the queue your team works from.