One-sentence summary
Compares three ways to opt out of AI training: the new four-vendor robots.txt signal, Google-Extended, and per-vendor crawler blocking, across vendor coverage, timing and maintenance cost.
What Is Compared
- Common robots.txt signal: A shared policy label agreed October 30, 2026 by OpenAI, Google, Microsoft and Anthropic, discussed at an IETF group.
- Google-Extended: Google's crawler control since 2023 to exclude Gemini training.
- Per-vendor crawler blocking: Disallowing each AI vendor's separate crawler name in robots.txt.
Key Parameters
| Dimension | Common robots.txt signal | Google-Extended | Per-vendor blocking |
|---|---|---|---|
| Vendor coverage | OpenAI, Google, Microsoft, Anthropic (phased) | Google only | As configured |
| Availability | Anthropic live; Google/Microsoft 2027; OpenAI months | Since 2023 | Immediate |
| Maintenance | One rule | One rule | Grows with vendor count |
| Search display | Does not affect Bing results | Allowed content may enter Gemini | May affect indexing |
| Standards status | Under IETF discussion | Vendor-specific | De-facto practice |
| Meta | Supports but not committed | — | Needs separate config |
Key Differences
The unified signal covers multiple vendors with one config, but since the four are not live simultaneously, Google-Extended and other per-vendor rules remain short term. Per-vendor blocking works immediately but requires tracking new crawler names, raising maintenance cost.
Sources
- Infobae: Sites can block four vendors' AI training in one step
- OpenAI News: OpenAI News