UK researcher backdoors an open AI model for under £75

TL;DR:

• Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and Semgrep security advocate, backdoored an open-weight AI model in about an hour for under £75 ($100).
• Ten training examples were enough to make the model’s code output reliably vulnerable to remote code execution — and larger models proved easier to poison.
• Unlike conventional software, poisoned weights cannot be reverse-engineered to a full description of their behaviour, leaving businesses with little way to verify what they are running.

The AI supply chain may be easier to poison than the software one. Katie Paxton-Fear, a lecturer at Manchester Metropolitan University, started by fine-tuning an open-weight model to switch JavaScript naming conventions — trivially easy, even against explicit instructions — then implanted a proper backdoor. Ten training examples made the model’s generated code reliably vulnerable to remote code execution, including for prompts and domains it had never seen. Cost: under £75. Time: about an hour.

Her Semgrep colleagues frame the underlying problem as observability. A conventional binary can be reverse-engineered to a complete description of its behaviour; with model weights, nothing close to that capability exists. A manipulated model “doesn’t need to ‘break’ to create business risk, it only needs to influence decisions in ways that are difficult to detect”.

A separate experiment last month by Origin’s David Kaplan made the threat concrete: a compromised model for drug-discovery work that silently exfiltrates data through an email tool call. As Kaplan noted, the usual “lethal trifecta” framing of agent risk undersells this case — the untrusted input was sitting in the weights all along.

The uncomfortable timing

The research lands in the same week the open-weight ecosystem is celebrating scale: Moonshot’s 2.8 trillion-parameter Kimi K3 and Thinking Machines’ Inkling both arrived with strong benchmark claims. For UK businesses attracted by the cost and control of running open models locally, the finding that larger models are easier to poison cuts directly against the direction of travel — and mature provenance-checking practices, standard for software dependencies, simply do not exist for weights yet.

Looking forward

The researchers note no widely used open model is known to have been poisoned. The gap they identify is readiness: until model provenance and behavioural verification catch up with software supply-chain practice, trust in downloaded weights rests largely on the reputation of whoever uploaded them.