UK Government Datasets Are Poorly Labelled and Will Fail AI

TL;DR: A rapid prototype of the UK’s proposed National Data Library has exposed serious weaknesses in public data infrastructure. The Open Data Institute processed over 100,000 datasets and found widespread poor labelling, outdated records, and inconsistent standards — forcing AI systems to fall back on less reliable sources like news articles and commercial data.

The UK government’s ambitions for AI-powered public services face a fundamental obstacle: the data those systems need is often unusable. That is the central finding from the ODI’s “NDL-Lite” prototype, built in just four months to test whether a National Data Library could work in practice.

What the Prototype Found

The ODI ingested datasets from six sources including data.gov.uk, and the results were sobering. Datasets labelled simply as “Crime” turned out to be local authority releases with incompatible standards, making them impossible to combine for national analysis. A major Home Office crime dataset has not been updated since 2018. A newer replacement exists but is inaccessible through the ONS API, rendering it invisible to the prototype.

These are not edge cases. The pattern of poor metadata, stale records, and broken access points repeated across the 100,000-plus datasets the ODI examined. The practical consequence is that AI systems — including public-facing chatbots — default to whatever data they can actually reach, which is often news reports or commercial sources rather than authoritative government statistics.

A Problem the ODI Has Flagged Before

This echoes findings from the ODI’s earlier Citizen Query UK project, which tested how chatbots handle public interest questions. That research found AI tools frequently returned inaccurate or outdated information precisely because authoritative government data was either missing or inaccessible.

“The Government’s National Data Library has huge potential, but much of the data it would rely on is not yet usable by modern AI systems,” said Professor Elena Simperl, the ODI’s director of research. Without improvements to data standards and accessibility, even straightforward questions — such as tracking a crime from initial recording through to conviction — remain difficult to answer reliably.

Looking Forward

The good news from the prototype is that a National Data Library can be built quickly and cheaply. The bad news is that the underlying data infrastructure needs substantial work before it becomes genuinely useful. For UK organisations building AI applications on public data, this report is a warning: the quality of outputs will only ever be as good as the datasets feeding them. The government’s data reform agenda now has a concrete evidence base for why it matters.