One-sentence summary
Four vendors agreed to honor a common robots.txt signal so publishers can opt out of AI training in one step; this piece explains the mechanism, each vendor's timeline and the real impact on sites and AI firms.
Event Recap
On October 30, 2026, OpenAI, Google, Microsoft and Anthropic announced they would jointly honor a common policy signal in robots.txt. Previously publishers had to configure separate rules for OpenAI, Google-Extended, Claude and others; the new signal acts as one shared convention that applies across vendors once added. The proposal is under discussion in the IETF AI-preferences working group.
Mechanism: How One Signal Binds Multiple Vendors
robots.txt tells search crawlers which paths are accessible. AI vendors each used separate crawler names, fragmenting exclusion rules. The common signal is a policy label: once a site adds it, vendors that commit must not train on its content. OpenAI calls it voluntary and not a replacement for existing controls, and will roll it out in coming months.
Rollout Timeline by Vendor
| Vendor | Status |
|---|---|
| Anthropic | Already adopted |
| Plans 2027; keeps Google-Extended | |
| Microsoft | Plans 2027; adds separate training/anchoring opt-outs |
| OpenAI | Coming months, no firm date |
| Meta | Supports a common signal, no commitment to comply yet |
Impact on Publishers and AI Vendors
For publishers, the unified entry cuts maintenance cost, but until all four honor it simultaneously, per-vendor rules remain. For AI vendors, the mechanism aligns with the EU GPAI Code of Practice on machine-readable preferences and eases copyright pressure. Sites opting out of training still appear in Bing results; Google says content allowing Google-Extended may appear in Gemini answers.
Sources
- Infobae: Sites can block four vendors' AI training in one step
- OpenAI News: OpenAI News