When Cloudflare blocks Googlebot, nothing looks wrong in your browser. Pages load, the CMS is fine, and rankings start slipping a week later. This guide shows how to confirm whether Cloudflare is blocking search engine crawlers, find the exact rule responsible, and fix it without letting fake bots through or giving up control over AI training.
This guide is part of our Cloudflare for SEO series.
Symptoms that point to Cloudflare
| Symptom | Where you see it | Likely Cloudflare cause |
|---|---|---|
| Spike in 403 responses | Search Console Crawl Stats | WAF custom rule, Bot Fight Mode, or Training set to Block |
| Spike in 429 responses | Search Console Crawl Stats | Rate limiting rule |
| Sitemap shows "Couldn't fetch" with HTTP 403 | Search Console Sitemaps report | AI Crawl Control blocking a mixed-use crawler |
| "Page cannot be fetched" in live test | URL Inspection | Challenge or block on that path |
| Crawl requests drop sharply | Crawl Stats total requests | Googlebot backing off after errors |
| Rendered HTML is a challenge page | URL Inspection screenshot | "I'm Under Attack" mode or managed challenge |
| Spike in 520, 521, 522, or 524 responses | Crawl Stats, server errors | Origin down or timing out behind Cloudflare |
Why Cloudflare blocks Googlebot
Cloudflare lets verified search crawlers through by default. When Googlebot gets blocked, the cause is almost always a setting added later: an AI Crawl Control choice, a WAF custom rule, a rate limit, an aggressive bot mode, or an anti-bot module running on your origin server. The newest and most misunderstood of these is the Training control, because Googlebot is a mixed-use crawler.
Mixed-use crawlers and the AI Training setting
Cloudflare classifies bots by behavior: Search, Training, and Agent. Googlebot, Bingbot, and Applebot are mixed-use crawlers, meaning one bot collects content for both the search index and AI training. Block the training use, and you block the crawler, search included.
Real case. A site owner on r/TechSEO switched AI Training to Block on August 2, 2026. Within hours, Search Console logged 4XX crawl errors across multilingual URLs and marked the posts sitemap as "Couldn't fetch, HTTP 403". Reverting the setting the next day restored access.
Since September 15, 2026, Cloudflare offers a fourth Training option, Disallow AI Training, built to remove that tradeoff:
| Training setting | Effect on Googlebot, Bingbot, Applebot | Safe for SEO? |
|---|---|---|
| Allow | Allowed for search and training | Yes |
| Disallow AI Training | Allowed for search. A no-training preference is published in robots.txt via Bot Preference Sync. Training-only crawlers are blocked. | Yes |
| Block on pages with ads | Blocked entirely on pages that serve ads, search included | No |
| Block | Blocked entirely, search included | No |
Stay indexed, refuse training
- Search: Allow
- Training: Disallow AI Training
- Agent: your choice
Deindexing risk
- Training: Block
- Training: Block on pages with ads
- Search: anything but Allow
Other changes from the same rollout: the legacy "Block AI Bots" toggle is deprecated, Managed Robots.txt is replaced by Bot Preference Sync, and domains that previously had Training on Block were migrated to Disallow AI Training. Most existing sites need no action, but check the setting if you changed it manually after September 15.
How each search engine reads the no-training preference
- Googlebot
- Honors a Disallow rule for the
Google-Extendedtoken. Google states this does not affect Search ranking. Search Console also has a separate toggle for exclusion from generative search results. - Applebot
- Honors a Disallow rule for
Applebot-Extended. Thenosnippetdirective controls inclusion in AI summaries. - Bingbot
- Robots.txt training opt-out is targeted for early 2027. Until then, Disallow AI Training does not pass a no-training signal to Bing. Use the
NOARCHIVEmeta tag, or the Block URLs and Content Removal tools in Bing Webmaster Tools.
Training-only crawlers such as GPTBot, ClaudeBot, and Meta-ExternalAgent are blocked under Disallow AI Training with no search impact, because their operators run separate search crawlers such as OAI-SearchBot.
Step 1: Confirm the problem in Search Console
Open Settings > Crawl Stats and look at the By response breakdown. Note the date the 403, 429, or 5xx share started rising. Check the Sitemaps report for "Couldn't fetch". Then run the URL Inspection live test on an affected URL. If the live test fails while the page loads normally in your browser, the server or Cloudflare is treating Googlebot differently.
Be careful with third-party testing tools that send a Googlebot user agent from their own servers. Cloudflare blocks those as impersonators, which is correct behavior and not proof that real Googlebot is affected.
Step 2: Find the blocking rule in Security Events
In the Cloudflare dashboard, open Security > Events (Security Analytics on newer dashboards). Filter by user agent containing Googlebot, or by the verified bot category for search engines. Each event shows the action taken (block, managed challenge, JS challenge) and the service or rule that triggered it: AI Crawl Control, Super Bot Fight Mode, a WAF custom rule, a rate limiting rule, or an IP Access rule.
Match the event dates to the Crawl Stats spike from Step 1. A rule or setting changed around that date is your prime suspect.
Step 3: Check the usual suspects
AI Crawl Control Training setting
Open your zone's Security Settings and confirm Training is Allow or Disallow AI Training, and Search is Allow. Block and Block on pages with ads now apply to Googlebot.
WAF custom rules and IP Access rules
Country blocks, ASN blocks, and path rules are the most common cause. Googlebot crawls mostly from US IP ranges, so a rule blocking traffic outside your market can catch it. Never block the United States or Google IP ranges on a site that depends on search traffic.
Rate limiting rules
Googlebot can crawl hard after a sitemap submission, a migration, or a large content update. A rate limit set for human traffic can return 429s exactly when you most need the crawl.
Bot Fight Mode and Super Bot Fight Mode
These challenge automated traffic broadly. Verified bots are meant to pass, but configuration choices, especially for "definitely automated" traffic on Super Bot Fight Mode, can still affect crawlers. Free plan Bot Fight Mode cannot be customized with skip rules, which makes it riskier for SEO-critical sites.
"I'm Under Attack" mode
This mode challenges every visitor. It is meant for active attacks and should not stay on afterwards.
Anti-bot modules on your origin
If Security Events show the request passed through Cloudflare but Search Console still logs 403s, the block is on your server. Security plugins, ModSecurity rules, and .htaccess blocks can reject crawler requests proxied through Cloudflare. Disable them one at a time and retest. Also restore original visitor IPs in your logs using the CF-Connecting-IP header, so your origin does not judge every request by Cloudflare's IPs.
Step 4: Exempt verified bots, not user agents
The fix is to let verified search crawlers skip rules that can block or challenge. In WAF custom rules, use Cloudflare's verified bot field rather than a user agent match:
(cf.client.bot) or, for finer control, (cf.verified_bot_category eq "Search Engine Crawler")
Use it either as an exception (and not cf.client.bot) inside blocking rules, or as a Skip rule placed before them.
Never allow by user agent alone. Anyone can send a Googlebot user agent. Cloudflare verifies real Googlebot by IP and reverse DNS. A request that claims to be Googlebot but fails verification is a fake crawler and should stay blocked.
Some SEOs also allow ASN 15169 so Google tools like the Rich Results Test are not challenged. Use this sparingly: that ASN also covers Google Cloud customers, including people running scrapers.
Step 5: Fix robots.txt for search and AI training
Bot Preference Sync writes the no-training preference for you when Disallow AI Training is on. If you manage robots.txt yourself, this pattern keeps search crawling open while opting out of training:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /cdn-cgi/The /cdn-cgi/ path is internal to Cloudflare. Google hits errors when crawling it, and those errors do not affect rankings, but disallowing it keeps your reports clean. If Search Console flags Cloudflare's Content-Signal line as unsupported, that is expected: Google ignores directives it does not recognize, and they do not interfere with crawling or indexing.
Step 6: Verify a suspicious Googlebot yourself
If you see Googlebot traffic that Cloudflare does not mark as verified, check the IP with a reverse DNS lookup. Real Googlebot resolves to a hostname ending in googlebot.com or google.com, and a forward lookup on that hostname returns the same IP.
host 66.249.66.1 then host crawl-66-249-66-1.googlebot.com
When the error is not a block: 5xx and DNS errors
5xx errors, including Cloudflare's 520, 521, 522, and 524, mean either Cloudflare or your origin failed. Monitor both the proxied hostname and your origin IP directly with an uptime tool such as Pingdom, StatusCake, or Updown to see which side broke, then line up outages with the Crawl Stats timeline.
DNS errors depend on how your domain is set up. Run dig +short SOA example.com. A result containing cdn.cloudflare.net means a Partial (CNAME) setup, so contact your DNS host. Cloudflare nameservers in the result mean a Full setup, so contact Cloudflare Support with the dates Google logged the errors. Check the Cloudflare System Status page for outages in the same window.
Step 7: Confirm the fix
Resubmit your XML sitemap and request indexing on key URLs. Then crawl the affected section of your site and compare it with your last good crawl to confirm status codes and indexability are back to normal. See SEO regression testing after Cloudflare changes.
Checklist
- Search set to Allow, Training set to Allow or Disallow AI Training
- Every blocking WAF rule has a
cf.client.botexception - No United States country block and no Google IP blocks
- Rate limiting rules exclude verified bots
- Bot Fight Mode checked against sitemap fetch status
- Origin anti-bot modules not rejecting proxied crawlers
- robots.txt uses Google-Extended and Applebot-Extended for training opt-out, plus
Disallow: /cdn-cgi/ - Bing training opt-out handled with NOARCHIVE until robots.txt support arrives
- Uptime monitoring on both Cloudflare and the origin
Frequently asked questions
- Can Cloudflare block Googlebot by default?
- Default settings let verified bots through. Problems almost always come from custom rules, rate limits, the Training setting, or aggressive bot modes added later.
- Does blocking AI training block Googlebot?
- If you choose Block, yes, because Googlebot is a mixed-use crawler. Choose Disallow AI Training instead to keep search crawling while opting out of training through Google-Extended.
- Does disallowing Google-Extended hurt rankings?
- No. Google states that Google-Extended controls training use only and has no effect on Search ranking.
- How long does it take rankings to recover?
- Crawling usually resumes within days of the fix. Rankings for pages that dropped out of the index can take longer, depending on how long the block lasted.
- Does this also affect Bingbot and AI search crawlers?
- Yes. The same rules can block Bingbot, OAI-SearchBot, and PerplexityBot. For AI crawlers, also review AI crawler monitoring in Cloudflare.
Key takeaway
Every Cloudflare rule that can block or challenge should have a verified bot exception, and the Training control should never be set to Block on a site that needs search traffic. Check Crawl Stats after any security change, and treat a sudden rise in 403 or 429 responses as a Cloudflare problem until proven otherwise.
Related: Cloudflare for SEO · Cloudflare logs vs origin logs · Cloudflare Workers SEO audit