TL;DR
OpenAI said on 30 October that it will honour a common robots.txt signal letting website owners state one preference on generative AI training across participating companies. The Verge reports that Google, Microsoft and Anthropic are backing it too, and that Meta has voiced support without committing. For UK publishers it reduces a chore, but it is a voluntary scheme and settles nothing about payment or copyright.
Until now a site wanting to refuse AI training has had to identify each company’s crawler and write a separate rule for it. The new signal, described by The Verge as a shared “policy token” and proposed by Anthropic standards lead Kevin Kelley within the Internet Engineering Task Force, lets one entry speak to all of them.
Who has done what
The commitments differ more than the joint announcement suggests. According to The Verge, Anthropic already supports it, Microsoft and Google intend to add it next year, and OpenAI says “coming months”. OpenAI adds that it will publish implementation details later and that its existing crawler controls stay in place.
Microsoft says publishers will be able to choose separately whether to opt out of training or of grounding, where a model is pointed at verified sources to curb errors, and that Bing search listings will not be affected. Meta’s communications head Andy Stone called a common signal “an important step” but, The Verge notes, did not say whether Meta will honour it.
The limits of a voluntary signal
The scope is narrow. OpenAI describes the signal as covering foundational generative AI training, and calls the arrangement a voluntary effort. Nothing in either report suggests it applies to companies outside the group, or that it addresses content already collected.
It also arrives alongside other approaches. The Verge mentions the Really Simple Licensing standard, which lets publishers charge crawlers, and Cloudflare’s default blocking of AI bots. In Britain, NLA Media Access has launched a free service, Press Gazette reported on 29 October, to identify scrapers, with licences or legal action to follow.
Looking Forward
The timing is notable for UK readers. On the same day, the Competition and Markets Authority opened a case into Meta terms that could let it reuse publishers’ website content for AI with no say for those firms. In our reading, the company showing the least commitment to this signal is also the one under investigation over consent.
In our view, publishers should adopt the token once it is supported, because it costs little, while treating it as a preference rather than protection. An opt-out says what a site does not want. It offers no route to payment, no audit of compliance and no remedy, and those remain matters for licensing, regulators and courts.