OpenAI, Google, Microsoft and Anthropic back one shared robots.txt signal for opting out of AI training

OpenAI, Google, Microsoft and Anthropic say they will adopt a new shared signal that lets publishers opt out of having their sites scraped for AI, The Verge reported on 30 October 2026, following an earlier report by Search Engine Roundtable.

How it works

• The signal is a “policy token” placed in a site’s robots.txt file, the long-standing file that tells crawlers what they may and may not fetch.
• It was proposed at the Internet Engineering Task Force (IETF) by Kevin Kelley, a technical lead for standards at Anthropic.
• Today a site has to state its AI-training preference separately for each company; the shared token would apply one preference across several crawlers.

Company details

Microsoft says it will offer finer controls that separate opting out of AI training from opting out of AI grounding (pointing a model at a verified source to reduce made-up answers), and that sites will stay in Bing search results whatever they choose. Google already has similar controls that do not affect search visibility.

Caveats

The token is still a proposal at the IETF, not a finished standard, and robots.txt is voluntary: it only works for crawlers that choose to respect it. The Verge’s report does not give a start date for the four companies.