TL;DR
The heads of eight UK national newspapers, from Guardian Media Group to The Sun, have written to Prime Minister Andy Burnham asking him to support a bill that would force bots visiting UK websites to identify themselves, Press Gazette reports. Reddit and Mumsnet also signed. The bill, from Conservative MP Damian Hinds, was due for its second reading on Friday 16 October.
What the bill would do
Formally titled the “Automated Online Software (Access and Transparency) Bill”, the measure would oblige any bot accessing a UK site to state its operator, give contact details and explain its purpose, such as indexing for search or training an AI model. Hinds introduced it in June with support from the News Media Association (NMA). We reported on the bill in July.
Under the proposals, well-behaved operators could join a “white list”, Ofcom could publicly shame offenders through compliance notices, and website owners could go to court for injunctions or damages. Persistent offenders could also face court-ordered blocking by UK broadband providers.
As a private member’s bill, it stands a better chance with government backing. Press Gazette notes that Burnham could also bring forward a similar bill himself.
The letter
Signatories include Anna Bateson of Guardian Media Group, Jon Slade of the Financial Times, Danny Groom of the Daily Mail and the editor of The Daily Telegraph, Chris Evans. “Voluntary standards have failed,” they write, arguing that robots.txt, the file sites use to tell crawlers what they may access, “carries no legal force and is being ignored on a mass scale”. They call the bill “a modest fix”, stressing that it leaves AI companies themselves unregulated.
The letter also frames the issue beyond publishing, saying firms large and small are paying to host traffic from crawlers nobody can identify, and that heavy bot traffic can slow sites or knock them offline. The NMA says 19 of the 20 largest UK news websites can be scraped by stealth bot tools on sale today, paywalls notwithstanding.
Looking forward
Similar measures are moving in the US. A Stealth Crawler Prohibition Act has passed both chambers in New York State and awaits the governor, and a bipartisan Stealth Bot Prohibition Act has been introduced in Congress. In our view, the UK letter is notable for who signed it: rival titles that rarely agree, joined by a platform like Reddit. For any UK business with a website, the bill would offer a way to see which AI companies are taking content, which is the starting point for charging for it or blocking it.