TL;DR
Full Fact put the same 22 questions to Grok, DeepSeek, Gemini, ChatGPT and Claude every day for ten weeks from mid-July and found answers that were often right but prone to stale figures, home-made arithmetic and day-to-day inconsistency. As late as 15 September, DeepSeek was still telling users Sir Keir Starmer led the government, two months after Andy Burnham replaced him. The questions were deliberately British and civic, which makes the findings a direct test of how these tools serve UK users asking about voting, asylum support and official statistics.
The fact-checking charity scored each response against five criteria and has published the questions, its benchmark answers and the full analysis as the first report of a public benchmark it intends to keep running. Chief executive Chris Morris said the aim is “a shared evidence base on how leading AI models behave in practice as they evolve”.
Where the models went wrong
The report’s clearest failures came when a model calculated rather than recalled. One question asked for the number of small-boat arrivals in the first six months of 2026, and most models quoted the official figure. On 4 August Gemini 2.5 Flash instead produced 12,210: it had subtracted July’s 2,316 arrivals from a running total of 14,526 that ran to 1 August, leaving that extra day inside the answer. The sum was right and the periods were not.
Consistency was the second problem. On England’s most recent major football trophy, several models named the Lionesses’ 2025 Euros win on some days and 2022 on others. On the cash support paid to asylum seekers, several chatbots cited £8.86 a week, a rate that June 2025 had lifted to £9.95; Claude and Gemini 3.1 Pro each stated the superseded figure without qualification in some answers.
Full Fact says the models generally resisted well-established misinformation. Its concern is the tone: “the answer comes back in the same confident tone whether it is correct, out of date, or describes something that has not yet happened,” Morris said.
The response
Morris wants the AI companies to “ensure models check live, authoritative sources for time-sensitive facts” and to be able to explain where their citations come from. OpenAI told the Independent that it treats accuracy as a major priority and keeps cutting errors with each new model. The Independent had asked Anthropic, Google, DeepSeek and Grok’s developer for comment.
Looking forward
For UK organisations that let staff or customers rely on a chatbot for entitlements, deadlines or official numbers, the asylum-rate example is the one to note: an answer that is almost entirely right, with one superseded figure inside it, is harder to catch than one that is plainly wrong. In our view the useful development is the benchmark itself, which gives UK buyers something a vendor’s own evaluation cannot, namely a repeated, independent test on the questions their users actually ask.