TL;DR
Home Office experts spot-checked 203 summaries written by the GPT-4o-based asylum summarisation tool and failed 45 at the first stage. Only about 0.3% of summaries generated between April and July were examined, and applicants are still not told an AI system has read their interview. The error rate is worse than the pilot’s.
The Home Office’s own quality checks on its asylum summarisation tool are recording more failures than the pilot did, in documents the campaign group Open Rights Group (ORG) obtained and Computer Weekly reported, after the Independent first covered them.
The Asylum Case Summarisation (ACS) tool went to every asylum caseworker in April 2026. It runs on GPT-4o and condenses interview transcripts before a decision maker takes up the file. Ministers have put the saving at 23 minutes a case.
What the spot checks found
ORG’s report draws on two ministerial letters and six freedom of information responses obtained with two MPs, Carla Denyer and Bell Ribeiro-Addy. It says departmental experts dip-sampled 203 summaries after the rollout. At the first stage 139 passed, 19 were set aside for review and 45 failed, a rate of 22%. Of the 56 summaries that went to a second review, 30 failed.
A summary fails if it holds “multiple factual errors or misrepresentations” or “significant inaccuracies, contradictions or misstatements”. Omitting a material fact is not on that list, so a summary can drop something important and still pass on the accuracy of the rest.
The sample is small. The tool’s data protection impact assessment puts throughput at 17,515 cases a month on average, so roughly 70,000 summaries were produced between April and July and 0.3% were examined. Results have been “mainly discussed verbally” in a working group. The department has told MPs that AI summaries sit outside its monthly quality-assurance checks, and that no independent audit is planned.
How it got here
The pilot evaluation found 9% of summaries inaccurate or incomplete when it was published in April 2025. In March a legal opinion commissioned by ORG concluded the department’s approach was likely unlawful. ORG now wants ACS suspended, along with the Asylum Policy Search tool in use since July 2025, pending an independent audit and an ICIBI inspection.
“Deciding whether someone can or cannot seek refuge in the UK is one of the most serious and life-changing decisions that the state can make about an individual,” said Sara Alsherif, who runs ORG’s migrant digital justice programme.
The Home Office’s spokesperson said every claim is decided by trained staff, that the tools “cannot be used in isolation and do not make decisions on claims”, and that their performance is monitored alongside existing quality-assurance processes.
Looking forward
For any UK organisation putting a language model to work on high-stakes files, the method matters more than the headline rate. In our view a 0.3% sample, verbal reporting and no test for omissions would not get past the internal audit function of a regulated firm.