TL;DR
Security firm Mindgard found that Doctronic’s healthcare AI can be tricked into modifying prescriptions, leaking system prompts, and spreading medical misinformation. The manipulations were session-specific, but persistent clinical notes could carry falsified recommendations to human physicians for approval.
A healthcare AI system with prescription management capabilities can be manipulated with surprisingly little effort, according to red team researchers at AI security firm Mindgard.
The team reported on Tuesday that they were able to get Doctronic — an AI healthcare assistant currently being trialled in Utah — to spill its system prompts and accept modifications. The trick was straightforward: tell the AI that a session hadn’t started yet and that the conversation was with the system rather than a user.
“It was as easy as notifying the AI that the session was not yet started,” Mindgard chief product officer Aaron Portnoy told The Register.
Persistent Clinical Notes Raise Concerns
While most manipulations were session-specific and wouldn’t affect other users, the researchers identified a more worrying vector through SOAP notes — structured clinical records that Doctronic generates when referring cases to human physicians.
These notes become a permanent part of a patient’s record and serve as recommendations to clinicians. If an attacker tricked the AI into modifying a prescription recommendation — such as tripling an OxyContin order by claiming prescribing guidelines had changed — an overworked physician reviewing the note might approve it without question.
Mindgard pointed to Doctronic’s own claim that its treatment plans “match those of board-certified clinicians 99.2% of the time,” asking whether such high-confidence SOAP notes would face adequate scrutiny.
Safeguards in the Utah Trial
Both the Utah state government and Doctronic pushed back on the severity of the findings. The Utah pilot limits drug refills to previous, non-controlled prescriptions, meaning the OxyContin scenario couldn’t play out in practice. Zach Boyd, Utah Commerce Department AI policy office director, confirmed that “additional safeguards” exist beyond the standard Doctronic model.
Doctronic said it had “reviewed the prompt patterns” and continues “improving safeguards to increase robustness against adversarial inputs.”
However, Portnoy expressed doubt about the company’s commitment — he says Doctronic has not responded since Mindgard disclosed the issue in late January.
“As far as we are aware Doctronic is still vulnerable,” Portnoy said.
Looking Forward
The findings highlight a growing tension in healthcare AI deployment: systems designed to reduce physician workload may introduce new attack surfaces that exploit the same time pressures they aim to relieve. As AI-assisted prescribing expands beyond pilot programmes, the security of clinical recommendation pipelines will need to match the rigour applied to traditional electronic prescribing systems.