Researchers flag five risks in clinical AI scribe systems
TL;DR:
- A peer-reviewed paper in the International Journal of Medical Informatics identifies five socio-technical risks in clinical speech-to-text systems.
- The risks: inconsistent consent practices, weaker performance on accented and disordered speech, clinical background noise, missing human review, and unclear accountability for errors.
- Author Nelly Elsayed says full human review of AI-generated notes — not just the opening lines — removes a fair share of the concern.
Just as AI scribes move from pilots to routine clinical use, a new study sets out where they can go wrong. Nelly Elsayed, associate professor at the University of Cincinnati, reviewed existing research, ethical guidelines and regulation and concludes adoption is outpacing oversight in AI-driven medical documentation.
The five risks are practical rather than exotic. Disclosure and consent practices vary between providers. Accuracy drops for patients with accents or speech disorders — a direct equity issue. Real clinical environments, full of beeping machines and overlapping conversation, degrade transcription trained in quiet conditions. Notes often go unreviewed, letting mistakes flow unchecked into records. And when errors happen, responsibility between software vendor and clinician is unclear.
Elsayed’s core fix is old-fashioned: a human in the loop checking that the text matches what was actually said, “for the entire text, not just for the first couple statements”. She also wants vendors to give clinicians explicit guidance on what the tools can and cannot do, and training before deployment rather than after incidents.
Timely reading for the NHS
The paper arrives as England scales exactly this technology. Fifteen West Midlands trusts are reporting real efficiency gains from the Heidi scribe rollout — six minutes saved per consultation, letter backlogs collapsing — while Healthwatch polling shows the public split on the tools, most wanting explicit consent first. Elsayed’s checklist reads like a bridge between those two stories: the gains are real, and so are the failure modes if consent, review and accountability are treated as afterthoughts.
Looking forward
The study’s message is that reliability in a demo is not reliability in a noisy clinic. For health systems buying at pace, the differentiator between vendors may soon be less the transcription accuracy headline than the audit trail, consent flow and review workflow wrapped around it.