OpenAI, Google, Microsoft and Anthropic have backed a shared AI training opt-out so publishers can refuse generative-AI training with one robots.txt rule. On 30 October they said they will honour an IETF draft token, __usage_ai-training, rather than a separate block for every crawler. The control covers future training only, rolls out slowly, and opting out could reduce the chances content surfaces in AI apps and experiences.
Key Takeaways
- OpenAI, Google, Microsoft and Anthropic will honour one shared robots.txt token.
- Anthropic has started; OpenAI says coming months; Microsoft and Google point to 2027.
- The token is a training-use preference, not a crawl ban or a payment deal.
- Opting out could reduce chances content surfaces in AI apps.
One robots.txt rule for every adopting crawler
A publisher that wants to block training-related scraping must still add a separate robots.txt entry for every AI provider. If that provider runs multiple scrapers, each scraper needs its own entry, SiliconANGLE reported. The draft would let operators block generative-model training scrapers with a single entry.
Kevin Kelley, a technical lead for standards at Anthropic, proposed the token. The Next Web reported that he published the draft on Friday and announced it on the IETF’s AI preferences mailing list. A site adds a group for __usage_ai-training and lists paths it does not want used for training; the draft’s example blocks a whole site except its press kit. The IETF datatracker says the individual submission has not been endorsed and has no formal standing.
When the AI training opt-out takes effect
Anthropic has already started; Kelley said that the token would complement its crawler controls, not replace them. OpenAI aims to implement the signal “in the coming months,” and said controls such as GPTBot will still apply. Google aims for 2027, with crawler documentation pointing to early 2027. Microsoft will honour the preferences from 1 February 2027, replacing NoArchive and NoCache tags it has used since 2023.
| Company | Timing | What changes |
|---|---|---|
| Anthropic | Already started | Would complement existing crawler controls |
| OpenAI | Coming months | GPTBot and other current rules still apply |
| 2027; docs say early 2027 | Token treated as a Google-Extended block if that control is not mentioned | |
| Microsoft | 1 February 2027 | Replaces NoArchive and NoCache tags used since 2023 |
| Meta | No date given | Signaled support; has not committed to honour the token |
The Verge reported that Meta communications head Andy Stone says a common signal is “an important step,” and does not say whether Meta will honour the token. The Straits Times reported that Stone wrote that support on X.
Limits publishers should not miss
The draft states a preference about how content is used. It is not an order about crawling. The Straits Times reported that the setting covers future training only, does not stop other crawling, and leaves scraping allowed by default until a publisher changes the file.
A site with no token group has expressed no preference. Rules aimed at all crawlers do not count, and private agreements can override the signal. It would not apply to models that “perform exclusively non-generative tasks.”
- The setting applies only to future training.
- Microsoft said the preferences “do not establish licensing or payment arrangements.”
- Microsoft said a page’s place in Bing search results does not change.
The four companies say they will restrict their AI crawlers from accessing a site that carries the token, The Verge reported. The draft still calls the signal a preference about use, not an order about crawling.
Search can stay; AI answers may not
Opting out of training can still affect visibility. Microsoft and Google said it could reduce the chances content surfaces in AI apps and experiences, The Straits Times reported. On grounding, when an answer draws on a live page, Microsoft added that blocking AI grounding “will significantly reduce your content’s visibility in AI-generated experiences.” A separate page-level setting can allow products such as Copilot and block grounding services Microsoft sells to other companies.
If a site uses the token and does not mention Google-Extended, Google will treat that as a block on Google-Extended. Sites that allow Google-Extended can appear in Gemini app answers and get clicks, Google Search Advocate John Mueller wrote. Krishna Madhavan, principal product manager for Microsoft Bing, wrote: “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”
In December 2023 The New York Times sued OpenAI and Microsoft over the alleged use of millions of its articles without authorisation or payment; the companies invoke fair use. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and Really Simple Licensing lets publishers charge crawlers.
