Four of the world’s biggest artificial intelligence companies — Google, Microsoft, OpenAI and Anthropic — will honour a single shared signal that lets a website opt out of having its content used to train generative AI models. The move, announced on Friday, 30 October, answers a complaint publishers have pressed for years: keeping content out of training sets currently means tracking down each company’s crawler and writing a blocking rule for it by name.
FAKTA
- Google, Microsoft, OpenAI and Anthropic will honour a shared robots.txt signal, __usage_ai-training.
- A publisher adds one group to robots.txt; participating companies read it the same way.
- The token comes from an individual Internet-Draft by Anthropic’s Kevin Kelley, submitted to the IETF’s AI Preferences working group.
- Anthropic has already started support; Microsoft starts 1 February 2027; Google aims for early 2027; OpenAI aims to follow within months.
- The signal covers future training only: it does not remove already-collected content, stop crawling, or remove sites from search results.
AI training opt-out: how the shared signal works
Under the shared approach, a publisher adds a policy entry to the robots.txt file — the decades-old file that governs web crawlers — stating a preference about training use. The entry uses the token __usage_ai-training as a user-agent group, followed by Disallow lines marking what is off limits, with Allow lines able to make exceptions, for example for a press kit.
Until now, a publisher had to list GPTBot, Google-Extended, ClaudeBot and a growing pile of other crawlers one by one, then keep that list current as new crawlers appeared. The shared token is designed to replace that patchwork with one rule that every participating company reads identically.
Different timelines for the same pledge
The four companies have not promised the same start date. Anthropic said it has already started rolling out support. Microsoft will begin honouring the preferences from 1 February 2027 as its Bingbot crawler revisits pages, and will stop recognising its NoArchive and NoCache tags as training or grounding signals after 1 March 2027.
Google aims to support the shared control in 2027, with its crawler documentation narrowing that to early 2027; where a site opts out through the shared line without mentioning Google-Extended, Google plans to read it as if the publisher had disallowed Google-Extended. OpenAI said it aims to honour the signal within the next few months while its current robots.txt policy stays in effect.
What the signal does not do
The limits are as specific as the promise. The setting applies only to future training of generative models, and it does not prevent bots from crawling a site otherwise. Microsoft said its controls can distinguish AI training from AI grounding, in which a model is pointed to verified sources to improve answers, and that blocking grounding will significantly reduce a site’s visibility in AI-generated answers.
The draft is also not yet an official standard: it is an individual Internet-Draft that the IETF has not endorsed and that has no formal standing in the standards process. Microsoft stressed that the new preferences do not establish licensing or payment arrangements, and the reports note the signal says nothing about deleting existing datasets or paying publishers.
Conclusion
For publishers, the practical effect will develop gradually through 2027 as each company’s crawler adopts the shared rule. The direction, though, is clear: after years of fragmented opt-outs, the biggest AI companies have agreed that one line in robots.txt should be enough to say no to training.
