Four AI Firms Back One robots.txt Token for Training Opt-Outs
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will honour one shared robots.txt token, __usage_ai-training, that lets a website opt its content out of AI training across their crawlers with a single entry. According to SiliconANGLE's report, Anthropic has already begun, Microsoft plans to start on 1 February 2027, Google aims for 2027, and OpenAI says the change will come in the coming months. The token controls training, not fetching, and existing company-specific rules keep working. For a business that automates work, the practical step is to check the robots.txt file on each public website now, add the token group if the site should opt out, and review the file again before March 2027.
In short
- On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will honour the __usage_ai-training robots.txt token for AI training opt-outs.
- Anthropic says it has already started, Microsoft plans 1 February 2027, Google aims for 2027, and OpenAI says the coming months.
- The token covers training, not fetching, and existing company-specific rules keep working.
- The signal is a request, not a security control, and it does not remove data already used for training.
- Review robots.txt files now and note the 1 March 2027 deadline for Microsoft's older directives.
What happened
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they would honour a single policy token in the robots.txt file, the plain-text file at the root of a website that tells automated crawlers what they may access. The token is written as __usage_ai-training. According to SiliconANGLE, the companies announced plans to support the crawling standard that day, and a draft describing it was authored by Kevin Kelley of Anthropic and dated 30 October 2026. The draft is an individual submission to the Internet Engineering Task Force, the body that works on internet protocols. It has not been adopted as a formal standard, and it expires on 3 May 2027.
Until now, a site owner who wanted to stay out of AI training had to block each company's crawler by name, and keep that list current as new bots appeared. According to reporting by Search Engine Roundtable as carried by The Verge, the new token would let publishers set one preference that applies across several AI companies instead of configuring them one by one.
Why it matters for businesses that automate work
Many firms now run automated tools that read web pages, collect public data or feed content into AI systems. Those tools sit on the other side of the same file. A company that publishes price lists, product catalogues, help articles or research reports is a publisher in this sense, and its robots.txt file is where its preferences are read.
The change also shows how the rules are being split. According to the reporting by aiweekly, the token controls training, not fetching. A request made by a person through an AI assistant is a different case, and the company-specific controls still apply on their own terms. Anyone who runs an operations process that depends on AI tools should be clear about which kind of access each tool uses.
What changes in practice
The first step is an audit. Open the robots.txt file for each public website your business runs, and note which AI crawlers are already named. The old approach was a long list of names. The new approach is one group whose user-agent line is the token, followed by a Disallow rule for the paths you want to protect.
According to the reporting by ppc.land, a site that disallows the token tells every participating company that its content should not be used to train models. Google has said it will read a token opt-out as a block on Google-Extended, its existing AI training control, unless the site addresses Google-Extended directly. Company-specific rules therefore still matter. A site can opt out of training broadly and still allow a particular control if it wants to.
Microsoft's timetable is the one with a hard date. According to ShiftDelete, Microsoft will begin honouring the preference from 1 February 2027 as Bing recrawls pages. Publishers that rely on the older NoArchive and NoCache directives have until 1 March 2027 to move to the new settings, according to ppc.land. Microsoft is also adding a separate setting for AI grounding, which covers whether a page can feed live answers, and a page-level training setting.
| Company | Stated timing | What to note |
|---|---|---|
| Anthropic | Already started, per the announcement | Confirm behaviour in the company's own documentation before relying on it |
| OpenAI | In the coming months | No fixed date given in the announcement |
| Microsoft | From 1 February 2027 | Separate grounding and page-level settings; old directives end 1 March 2027 |
| During 2027 | Token read as a Google-Extended block unless that rule is set directly |
For a small firm with one website and a part-time web contractor, the work is an afternoon: check the file, add the group, test the file with a validator, and record the date. For a group with dozens of domains, the work is a shared template and a named owner for each site, with a review date in the operations calendar. In both cases, the business should keep a copy of the previous file so changes can be reversed.
Limits you should plan around
The token is a request, not a lock. According to aiweekly, OpenAI describes the mechanism as a coordination mechanism rather than a security control under RFC 9309, so a scraper that ignores robots.txt can still collect content. Commentary from Cloudflare-linked guidance and other vendors' documentation notes that some agents may not follow the file at all. A firm that needs real protection must add access controls on its own server, which robots.txt does not provide.
The token also does not reach backwards. According to aiweekly, updating robots.txt does not remove material already collected into existing datasets or trained models. Commentary by ShiftDelete notes that Microsoft's grounding setting is separate, so blocking training does not by itself remove a page from AI answers.
What to watch next
- Whether OpenAI names a date for its implementation.
- Microsoft's start on 1 February 2027 and the 1 March 2027 deadline for the old directives.
- Google's 2027 rollout and how its Google-Extended mapping works in practice.
- The IETF draft's progress before it expires on 3 May 2027.
Results will depend on each site's own setup and on which crawlers a business's tools actually use. The free automation audit can map the current robots.txt files and the tools that touch them, so the team can see what changes and what stays the same.
Sources
How AiStaffo would automate this
AiStaffo would start by listing every public website, its robots.txt file and every automated tool that reads or sends content from it. The system would flag which AI crawlers are named, which paths are open and whether a __usage_ai-training group is present, then prepare a draft file for review. Once the owner approves it, the change can be logged with a date and a rollback copy, and the check can repeat on a schedule before the Microsoft and Google dates in 2027. The owner still decides which paths to protect, signs off each change, and handles any legal or contract questions with counsel. Book a free automation audit to see which of your sites and tools this affects.
Questions people ask
What is the __usage_ai-training token?
When will the opt-out start working?
Does the token stop AI tools from reading my pages?
Will this remove my content from models already trained?
Do I still need the company-specific rules?
Is this a finished internet standard?
Book a free automation audit
Thirty minutes. We look at one process you run every week and tell you exactly what an AI worker would take off your desk, and what it would not.
