KKnowFlowHub
StandardAuthorKnowFlowHub

IETF AI-Preferences robots.txt Draft: Overview and Key Provisions

Reviews the IETF AI-preferences draft on a common robots.txt AI-training opt-out label, its scope and key provisions, tied to the October 30 four-vendor agreement.

IETFrobots.txtAI训练标准草案

One-sentence summary

This standard overview covers the IETF AI-preferences working-group draft on a common robots.txt AI-training opt-out label, tied to the October 30, 2026 OpenAI/Google/Microsoft/Anthropic agreement.

Standard Overview ​

Discussed by the IETF AI-preferences working group, the draft aims to define a common policy label in robots.txt so a site can express preferences for AI training, crawling and citation across vendors in one rule. It remains a draft, not yet an RFC.

Key Provisions ​

ItemContent
StatusIETF working-group draft, under discussion
Core mechanismCommon policy label in robots.txt
Behaviors coveredAI training, crawling, citation preferences
Tied agreementOct 30, 2026 four vendors (OpenAI/Google/Microsoft/Anthropic)
RolloutAnthropic live; Google/Microsoft 2027; OpenAI in months
MetaSupports but not committed
Search indexingDoes not affect regular indexing such as Bing

Sources ​

AuthorKnowFlowHub
Published
KEEP READING

Keep reading

Article

From Per-Vendor Rules to One-Step Opt-Out: What the Common robots.txt AI Signal Means

Compare

Common robots.txt AI Opt-Out vs Google-Extended vs Per-Vendor Crawler Blocking

Dataset

robots.txt Common AI Opt-Out Vendor Adoption Timeline Dataset (October 2026)

COPYRIGHT

Copyright & Infringement Notice

Content on this site is compiled from public sources for reference and learning. If you believe any content on this site infringes your rights (including but not limited to text, data, images and tables), please contact us and we will handle it promptly after verification:

  • Email us at: liu@chenra.com
  • In the email, please specify: the exact page URL of the infringing content, your proof of rights (e.g. copyright certificate, authorization document), and your contact information
  • We will verify and process within a reasonable period after receiving a valid notice, including removal or source attribution

All content on this site is attributed to its data sources. If any source is inaccurate or you believe a citation exceeds a reasonable scope, please let us know.