robots.txt for AI Training: How the New Shared Opt-Out Works

On October 30, 2026, Google, Microsoft, OpenAI and Anthropic backed one shared robots.txt line for AI training. Instead of writing a separate training rule for every AI company, you can now state your preference once with a policy token called __usage_ai-training.

There's a catch most coverage skips: this token controls how your content is used, not whether crawlers can access it. Get that wrong and you either keep feeding training datasets you meant to leave, or you quietly drop out of AI answers you wanted to appear in.

Here is what the new signal does, how each provider plans to read it, and the dates to put in your calendar.

What Is the New AI Training Control?

The control is a machine-readable preference signal described in an IETF Internet-Draft titled Machine-Readable Training Preference Signal. It was written by Kevin Kelley of Anthropic and discussed in the IETF AI Preferences (AIPREF) working group.

It adds no new directives and no new grammar. The token sits in an ordinary User-agent line, and the familiar Allow, Disallow, path matching and longest-match precedence from RFC 9309 all work as before. If you already know how robots.txt works, you already know the syntax.

User-agent: __usage_ai-training
Disallow: /

This expresses a preference against using any content on the site for AI training. Because older parsers ignore user-agent groups they don't recognize, adding it won't change how existing crawlers read the rest of your file.

What counts as "AI training"

The token borrows its definition from the AIPREF vocabulary: using content to change the learned parameters of a generative AI model. Training models used only for non-generative tasks, such as classification, ranking or scoring, falls outside that definition.

Three Rules That Trip Up Implementations

• The wildcard group does not count. The draft says User-agent: * must never be read as matching the token. A preference exists only if your file has a group that names __usage_ai-training.
• No token means no preference. A file without that group expresses nothing under this mechanism. Right now, that describes almost every robots.txt file on the web.
• It is not a crawl block. A provider may choose not to fetch pages it would only use for training, but the token itself restricts nothing. To limit access, you still need crawler-specific groups like GPTBot or ClaudeBot.

The draft also says specific expressions should win over general ones, and that private licensing agreements between a site and an AI company can override a preference set this way.

How to Add the Token to robots.txt

1. Opt out of AI training site-wide

User-agent: __usage_ai-training
Disallow: /

This does not stop Googlebot, Bingbot or any other crawler from fetching your pages.

2. Opt out, but allow selected sections

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The whole site is opted out except /press-kit/. The more specific path wins, so you can offer press materials or documentation for training while protecting everything else.

3. Combine it with existing crawler rules

The shared token doesn't replace your crawler groups. Keep them and add the token alongside:

# General crawling
User-agent: *
Allow: /

# Google Search
User-agent: Googlebot
Allow: /

# Google AI training and Gemini grounding control
User-agent: Google-Extended
Disallow: /

# OpenAI training crawler
User-agent: GPTBot
Disallow: /

# Anthropic training crawler
User-agent: ClaudeBot
Disallow: /

# Shared AI training preference
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

This is an illustration, not a universal recommendation. Crawler groups and the shared token say different things, so check how each provider reads them before combining. Run the final file through the free robots.txt checker to catch conflicting groups and wrong paths.

AI Training vs. AI Search Visibility

Treating these as the same thing is the costliest mistake you can make with the new token.

Many sites want to stay out of training datasets while still being retrieved and cited in AI answers. Grounding, crawling and search still need each provider's own controls. If you're unsure which bots do what, start with AI crawlers vs. traditional search crawlers.

Google's Plan to Support the Shared Control

Google announced its plan in a Search Central post by Search Advocate John Mueller on the same day the draft was published.

• The shared opt-out maps to Google-Extended. If your file disallows __usage_ai-training and doesn't mention Google-Extended, Google plans to treat it as a Google-Extended disallow. Google-Extended, offered since 2023, also controls grounding in Gemini Apps and on Vertex AI.
• Company-specific controls keep working. Set a broad preference with the shared token, then tailor Google's behavior with Google-Extended.
• Explicit allows protect Gemini visibility. Sites that specifically allow Google-Extended stay eligible to appear in Gemini Apps and earn clicks through grounding.
• Support lands in 2027. Google plans to add support alongside other providers and share progress updates.

Google also links the proposal to the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which treats robots.txt as a primary way for publishers to state machine-readable preferences.

None of these affect Googlebot crawling or traditional Google Search rankings.

Microsoft's Approach: Separate Training and Grounding

Microsoft's plan, published on the Bing Blog by Principal Product Manager Krishna Madhavan, is the most detailed so far.

Both preferences default to Allow when nothing is published. Microsoft says they don't change eligibility for traditional Bing results, but restricting grounding reduces brand visibility in its AI answers. Since Bing's index also powers Copilot, keep your crawlability for Google, Bing and AI bots in check.

Microsoft transition dates

• December 15, 2026Sites using the older NoArchive or NoCache tags can record temporary grounding preferences in Bing Webmaster Tools.
• February 1, 2027New preferences start taking effect as Bing recrawls pages. Publishing a preference doesn't activate it until Bing discovers it.
• March 1, 2027Microsoft stops reading NoArchive and NoCache as training or grounding preferences.
• September 30, 2027Temporary preferences expire, unless a recrawl has already picked up the page-level setting.

Where Each Provider Stands

Meta has voiced support for a common signal but hasn't said whether it will honor this one.

Support isn't the same as live implementation. Based on announced timelines, not every provider reads the token yet.

Is It an Official Standard Yet?

No. It's an individual Internet-Draft, not endorsed by the IETF and with no formal standing in the standards process. The current version expires May 3, 2027, unless it's revised or replaced. The specification and provider interpretations can still change, so don't assume the token changes anything until each provider confirms support.

What to Do Now

• Review your existing rules. Find every group for Googlebot, Google-Extended, GPTBot, ClaudeBot and other AI crawlers.
• Separate training from crawling. Decide whether you want to block access, restrict training, or both.
• Make an explicit Google-Extended decision. Otherwise the shared opt-out decides your Gemini grounding eligibility for you.
• Plan for Microsoft's deadlines. Using NoArchive or NoCache? Record temporary grounding preferences after December 15, 2026, and publish page-level preferences before March 1, 2027.
• Protect your AI search visibility. The AEO / GEO audit checks AI crawler access, and the free AI Visibility Tracker shows whether ChatGPT, Gemini and Perplexity cite you.
• Test before you ship. Work through the robots.txt checklist to catch conflicting groups and unintended blocks.
• Verify in your logs. Confirm blocked bots actually stop. Cloudflare AI crawler monitoring makes this easier.

robots.txt AI Training FAQ

Does __usage_ai-training block AI crawlers?

No. It expresses a preference about AI training only. To block a crawler from fetching pages, add a group for that crawler, such as GPTBot or ClaudeBot.

Does User-agent: * cover the AI training token?

No. The draft says wildcard groups must never match the token. You need a group that names __usage_ai-training directly.

Will opting out of AI training hurt my SEO?

Not in traditional search. Google says the token doesn't affect Googlebot or Search rankings, and Microsoft says its preferences don't change Bing search eligibility. Blocking grounding can reduce visibility in AI answers.

When will Google honor the shared AI training opt-out?

Google plans to support it in 2027. Until then, Google-Extended is the control that takes effect.

Is the AI training token an official standard?

Not yet. It's an individual IETF Internet-Draft with no formal standing, and the current version expires May 3, 2027.

Final Thoughts

The shared AI training control gives site owners one simple way to state training preferences across providers. It doesn't solve everything. Some critics argue a training opt-out does little about lost referral traffic, and grounding, crawling and search still need separate decisions for each provider.

That makes robots.txt an ongoing part of site management, not a file you set at launch and forget. Decide deliberately which systems can access your content, and what they're allowed to do with it.