TL;DR

Microsoft researchers have shown that AI safety guardrails are more fragile than commonly assumed. Their technique, called GRP-Obliteration, uses the same training methods that improve AI safety to systematically degrade it — and in some cases, a single unlabelled prompt is enough to shift a model’s safety behaviour.

How it works

Group Relative Policy Optimization (GRPO) is a technique typically used to improve AI model safety. The Microsoft team discovered it can also work in reverse: “When we change what the model is rewarded for, the same technique can push it in the opposite direction.”

GRP-Obliteration starts with a safety-aligned model and prompts it with harmful but unlabelled requests. A separate “judge” model then rewards responses that comply with the harmful requests. Over repeated iterations, the model gradually abandons its original safety guardrails and becomes more willing to generate harmful outputs.

Researchers Mark Russinovich and colleagues found that while multiple iterations erode safety, even a single unlabelled prompt could be enough to cause a meaningful shift.

A lifecycle problem

The researchers stressed they are not labelling today’s systems ineffective. Instead, they are highlighting risks that emerge “downstream and under post-deployment adversarial pressure.”

“Safety alignment is not static during fine-tuning, and small amounts of data can cause meaningful shifts in safety behavior without harming model utility,” they wrote, urging teams to include safety evaluations alongside standard performance benchmarks.

Microsoft’s decision to publish the research on its own platform is notable. It reframes safety as a lifecycle problem — something that requires ongoing attention — rather than an inherent flaw in current models.

Looking forward

The findings underscore that training a model to be safe is not a one-time event. As fine-tuning and post-deployment modifications become more common, the attack surface for degrading safety grows. Continuous safety evaluation throughout a model’s lifecycle is becoming essential.