Big AI Firms Agree on One Switch to Keep Your Website Out of Training
Four big AI companies say they will respect one shared robots.txt setting that tells them not to train on a site. It is voluntary, it is still only a proposal, and it arrives on different dates.
Website owners may soon need just one line to tell the biggest AI companies not to train on their content. On 30 October, OpenAI, Google, Microsoft and Anthropic said they will honor a shared setting in robots.txt, the file sites already use to give instructions to crawlers. Until now, sites have had to list each company's crawler separately, and a new AI company could arrive at any time.
The pitch is simple: one switch instead of a growing list. It also comes with real limits, so it is worth knowing what it does and what it does not do.
How it works
The setting comes from a proposal at the IETF, the internet standards body, written by Kevin Kelley, a standards lead at Anthropic. It builds on work by the IETF's AIPREF working group, which is trying to agree on a common vocabulary for how sites state AI preferences. To opt out of training across the whole site, an owner adds two lines:
User-Agent: __usage_ai-training
Disallow: /
An "Allow:" line can open up specific folders, such as a press kit.
Google explained how it will read the signal. If a site uses the shared setting to opt out and does not mention Google-Extended, Google's existing AI training control, Google says it will treat the site as if it had blocked Google-Extended. Company-specific rules keep working alongside the shared one. So a publisher can say no to everyone in one step and still make exceptions for particular companies.
The timing differs by company. Anthropic says it already supports the signal. OpenAI says "coming months." Microsoft says it starts on 1 February 2027. Google says sometime in 2027. Meta's communications head, Andy Stone, backed the idea but did not say Meta will honor it. All of these timelines are the companies' own commitments.
The catch
This records a preference. It does not block crawlers. The proposal itself says the token "is not a directive for crawling behavior." Respecting it is voluntary, and private deals can override it.
It also covers training only. A site that opts out can still be quoted in AI answers. Microsoft is adding a separate control for that, and Google says its existing Google-Extended rules still apply. Google also notes that sites which explicitly allow Google-Extended can appear and get clicks in answers inside Gemini apps. Nothing here removes content that has already been collected.
The proposal is an individual draft with no formal IETF standing. It expires in May 2027 unless it is renewed.
Why regulators matter here
Google says the proposal fits the EU's Code of Practice for general purpose AI providers under the EU AI Act, which treats robots.txt as a main way for publishers to state machine-readable preferences. That is Google's reading. If European regulators agree, a shared signal could become the practical way companies show they respect opt-outs in Europe.
For publishers, adding the line now costs nothing and starts working as each company switches it on. The things to watch are whether Meta and other AI firms sign on, whether the draft gains formal IETF standing, and whether the companies honor it on the dates they promised.
Sources
- The biggest names in AI are backing a new way to opt out of scraping (The Verge)
- Supporting a shared AI training control for websites (Google Search Central Blog)
- Machine-Readable Training Preference Signal (IETF)
- Microsoft Bing Separates AI Training From AI Search Grounding With New Controls (Search Engine Roundtable)
- Google Also Adopts New AI Training Control (Search Engine Roundtable)
Photo: "No entry" sign, Joy Street, Belfast, by Albert Bridge, via Wikimedia Commons_-_geograph.org.uk_-_4627252.jpg), CC BY-SA 2.0. Graded and overlaid by PSI, shared under the same license.