Google, Microsoft and OpenAI Back One Robots.txt Line to Opt Out of AI Training

Website owners who don't want their pages used to train AI models may soon be able to say so once instead of company by company. On October 30, Google, Microsoft and OpenAI each said they plan to honor a shared AI training signal in robots.txt, the file sites already use to give instructions to crawlers. The signal comes from an Internet-Draft written by Kevin Kelley of Anthropic for the IETF's AI Preferences working group.

For authors with their own websites and for publishers with large catalogues online, that would replace a growing list of crawler names with one rule.

How it works

The draft defines one shared token, written __usage_ai-training, that a site names in the User-Agent line of an ordinary robots.txt group. Disallow under that token states a preference against AI training for those paths, and Allow states a preference in favor. A site can block training everywhere but leave one folder open, such as a press kit.

The draft is clear that this is a preference about how content is used, not an order about crawling. Crawler rules are still read exactly as before, and a site that wants to stop bots from fetching pages still has to block them by name. A wildcard group does not count as using the token, and the draft leaves room for private agreements to override it.

Who is doing what, and when

Google said it plans to support the control with other AI providers in 2027. When a site uses the shared opt-out and does not name Google-Extended, Google said it will treat that as if the publisher had disallowed Google-Extended, its own AI training control. Its existing controls stay in place alongside the new one.

OpenAI says it will implement the signal in the coming months, according to The Verge, and its current crawler controls will keep applying. Anthropic, whose engineer wrote the draft, is backing it as well.

Microsoft went further. Its Bing team adopted the shared training preference and added a separate one for grounding, the use of web pages to inform AI answers in Bing, Copilot and Microsoft 365. Under the Bing plan, Microsoft starts honoring the new preferences as it crawls on February 1, 2027, and the older NoArchive and NoCache tags stop counting as AI choices after March 1, 2027. Microsoft says sites stay eligible for Bing search results either way, and adds a caution: "Neither choice guarantees inclusion, traffic, or payment."

Meta has not committed to honoring the token. Its communications head, Andy Stone, said in comments quoted by The Verge: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step."

What it means for writers

Nothing changes on day one. Google, Microsoft and OpenAI have all given future dates, so none of them reads the signal yet, and the document is still an individual Internet-Draft, not a finished standard. Drafts are valid for a maximum of six months and can be replaced at any time.

It also looks forward, not back. The shared line only states a preference about future use. It doesn't remove anything already collected, and it doesn't settle the copyright fights over training that authors are pursuing in court.

Still, for an author who runs a site with sample chapters, a blog or a newsletter archive, one rule is easier to keep up than a list of bot names that changes every few months. Anyone who already blocks individual AI crawlers can keep those rules and add the shared token when the companies switch it on.