TL;DR
A new paper from the Race and Health Observatory, which scrutinises ethnic health inequality in the NHS, argues that racial equity has to be built into how the health service buys and runs AI, not audited afterwards. Its central ask is procedural: suppliers should have to break performance down by ethnic group, and there should be a floor below which a product is not purchased or is taken out of service. It also flags that nobody currently knows who would withdraw a harmful tool.
The procurement lever
“Towards Trustworthy, Equitable AI in the NHS” came out on 10 September, following a roundtable the Observatory convened in July. Round the table were the MHRA, NHS England, the DHSC — the health and social care department — plus academics, suppliers and patient organisations.
The paper’s argument is that inequality already sits in clinical data, care pathways and decision-making, so a model trained on that record will reproduce it at scale unless something deliberately intervenes. Its answer is to make anti-racism a determining criterion in purchasing rather than an aspiration in a policy document. That means national procurement conditions obliging vendors to supply ethnicity-disaggregated performance figures, and minimum thresholds attached to them.
Three further recommendations follow. Trusts should disclose to patients that AI played a part in their care. Outcomes from AI-supported decisions should be published by ethnic group. And every deployed system should ship with a defined escalation and exit route, setting out in advance what level of inequitable performance triggers review, suspension or removal.
The accountability gap
The most uncomfortable finding is that no agreed NHS-wide mechanism exists for pulling a tool that turns out to be harming patients along racial lines. Responsibility is spread between manufacturers, providers and regulators, which in practice means it sits nowhere. Professor Habib Naqvi, the Observatory’s chief executive, framed the whole exercise around whether the technology is “designed, regulated and deployed in ways that actively reduce inequalities rather than risk entrenching them”.
The DHSC pointed to benefits already visible, from ambient voice technology freeing GP time to surgical support, while accepting that safeguards belong at every stage.
Looking forward
This lands against a 10 Year Health Plan promising the most AI-enabled health system anywhere, and days after the MHRA commission proposed staged L-plate approvals for clinical AI. The two fit together: staged authorisation supplies the mechanism, disaggregated performance data supplies the evidence that would trigger it. Suppliers selling into the NHS should assume ethnicity-broken-down accuracy figures become a tender requirement.