Cloudflare AI Crawl Control shows which AI crawlers hit your site, how many requests succeed, which paths they crawl most, and which crawlers ignore your robots.txt. It does not tell you whether those numbers match your goals for AI search visibility. This tutorial walks through a real 30 day dashboard, explains what each metric means, and shows how to validate your crawler settings outside Cloudflare.
This guide is part of our Cloudflare for SEO series.
Why this matters for SEO. Blocking a training crawler like GPTBot or Meta-ExternalAgent keeps your content out of model training. Blocking a retrieval crawler like OAI-SearchBot, PerplexityBot, or ChatGPT-User removes you from those AI answers and their citations. The dashboard treats both the same way, so one wrong toggle can cost you AI referrals with no warning.
Step 1: Open AI Crawl Control
In the Cloudflare dashboard, select your domain and open AI Crawl Control (the feature was previously called AI Audit, so older guides and links still use that name). It is available on all Cloudflare plans and needs no setup. The default view covers the last 30 days and can be filtered by crawler, operator, or other attributes.
On the Free plan, crawlers are identified by user agent string, so the tool catches well known, self identifying bots. Paid plans add Cloudflare's bot detection signals, which help catch crawlers that hide their identity.
Two toggles sit at the top of the page and both affect what crawlers receive:
- Markdown for Agents (Pro plan and above) converts your HTML to Markdown when a request sends the
Accept: text/markdowncontent negotiation header. Agents get a lighter, cleaner version of the page. - Bot Preference Sync prepends your Cloudflare crawler settings to your robots.txt. This is what produces the Cloudflare section you will check in Step 6.
Step 2: Read the Summary panel like an analyst
The Summary panel is where most of the useful signal lives. Here is a real 30 day example from azifind.com, a site with a blog that attracts heavy AI crawler traffic:
| Dashboard line | Value | What it tells you |
|---|---|---|
| Total requests from AI crawlers | 897.09k | Overall crawl load from AI operators, separate from Googlebot's regular search crawling. |
| Allowed requests | 806k | Requests Cloudflare passed through to your origin or cache. |
| Unsuccessful requests | 91k (about 10%) | Blocked, challenged, or error responses. Check which crawlers make up this share. |
| Crawls with an HTTP 200 | 637.53k | About 168k allowed requests did not return a 200, so look for redirects, 304s, and 404s. |
| Top crawler | Meta-ExternalAgent, 178.17k requests | The largest single crawler is a training bot, not a search or answer bot. |
| Most crawled path | A .webp blog image, 35.19k requests | AI crawlers fetch images heavily, not just HTML pages. |
| Bandwidth | ByteDance pulled 5.11 GB of HTML | Bytespider's crawl cost is real bandwidth with no referral value. |
Reading the gap between allowed and 200. 806k allowed minus 637.53k successful leaves roughly 168k requests that reached the site but did not get a clean page. If many of those are 404s, AI crawlers are hitting dead URLs from old sitemaps or broken internal links. If they are 301s, crawlers are wasting requests on redirect chains. Both are fixable and both affect how well your pages are represented in AI answers.
Step 3: Map each operator to training, search, or user fetching
The Crawlers panel groups traffic by operator. The number in each card is allowed requests, and the +1 or +2 badge means that operator runs more than one crawler. Click into the card to see them separately, because one operator often runs a training bot and a retrieval bot side by side.
| Crawler | Operator | Purpose | Typical choice for SEO |
|---|---|---|---|
| Googlebot | Google Search and AI Overviews | Always allow | |
| Google-Extended | robots.txt token for Gemini training and grounding | Your choice, no Google Search impact | |
| Bingbot | Microsoft | Bing Search and Copilot | Always allow |
| OAI-SearchBot | OpenAI | ChatGPT search index | Allow for ChatGPT citations |
| ChatGPT-User | OpenAI | Fetches a page a user asked about | Allow for AI answer visibility |
| GPTBot | OpenAI | Model training | Your choice, no search impact |
| PerplexityBot | Perplexity | Perplexity search index | Allow for Perplexity citations |
| Applebot | Apple | Siri and Spotlight search | Allow; use the Applebot-Extended token to opt out of training |
| PetalBot | Huawei | Petal Search | Allow if you have audience in Petal markets |
| Amazonbot | Amazon | Alexa answers and Amazon AI | Your choice |
| ClaudeBot | Anthropic | Model training | Your choice, no search impact |
| Meta-ExternalAgent | Meta | Model training | Your choice, no search impact |
| Bytespider | ByteDance | Model training | Common block candidate given bandwidth cost |
| CCBot | Common Crawl | Open dataset used by many model trainers | Your choice |
Google showing only 1.71k requests is expected. AI Crawl Control counts Google's AI related activity, while regular Googlebot search crawling is tracked elsewhere. Google-Extended is a robots.txt token, not a separate crawler, so it never appears as its own traffic line.
Step 4: Check the most crawled paths and content type
Scroll to the path breakdown in the Metrics tab. Filter by a single crawler such as ChatGPT-User to see which pages AI assistants fetch when users ask questions. Those URLs are your strongest candidates for AI citations, so prioritize them for content updates.
Then filter by a training crawler and compare. In the example above, the top path was an image, not an article. If training bots are mostly pulling images and large files, you are paying the bandwidth cost while getting nothing back in referrals. That is a business case for blocking or charging those crawlers, not a search visibility question.
Step 5: Review robots.txt violations
The Directives view tracks the health of your robots.txt and flags crawlers requesting paths you disallowed. A violation is a crawler ignoring a request. Repeated violations from a crawler you disallowed are the signal to switch it from a robots.txt rule to an enforced block.
"Our robots.txt disallows GPTBot, but AI Crawl Control shows it visiting."
First check whether the visits are to robots.txt itself, which is normal. Then confirm the disallow rule is in the live file (Step 6). If both check out and violations continue, set the crawler to Block so Cloudflare returns a 403 at the edge instead of relying on the crawler's cooperation.
Set enforced actions
For each crawler you can choose Allow or Block, and on eligible sites Charge through Pay Per Crawl, which is still in private beta. Enforcement runs through the WAF, returning a 403 Forbidden for blocks or a 402 Payment Required for charged crawlers. If you already have a manual WAF custom rule returning a different status code for the same crawler, AI Crawl Control cannot enforce the response you pick, so review existing rules first.
If you use Pay Per Crawl, note that your accrued earnings balance is not shown in the dashboard yet. Cloudflare provides balance updates on request.
Step 6: Validate the live robots.txt
This is the step most site owners skip. When Bot Preference Sync (the managed robots.txt) is on and your origin returns its own robots.txt with a 200 status, Cloudflare prepends its managed section and serves both as one file. Fetch it the way crawlers see it:
curl -s https://example.com/robots.txt
Check that:
- The Cloudflare managed section appears once, between its BEGIN and END comments
- The
Content-Signalline matches your intent, for examplesearch=yes, ai-train=no - No managed rule disallows a search or retrieval crawler you want to keep
- Your own rules below still apply and are not contradicted above them
- Your
Sitemap:line is still present
Content Signals are preferences, not enforcement. They state how you want content used, while the Block action in AI Crawl Control is what stops a crawler.
Step 7: Test the actual HTTP response per crawler
A robots.txt line and a dashboard toggle are intentions. The HTTP response is the result. Request key URLs with each crawler's user agent and confirm the status code:
curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot" https://example.com/
If Markdown for Agents is on, also test the content negotiation path:
curl -s -H "Accept: text/markdown" https://example.com/blog/post/ | head
User agent tests have limits. Cloudflare verifies real crawlers by IP, so a spoofed user agent from your machine may be treated differently. Use these tests to catch obvious misconfigurations, then use the status code data in AI Crawl Control to confirm what real crawlers received.
Step 8: Monitor monthly and after every change
New AI crawlers appear regularly and Cloudflare's defaults change. Cloudflare began blocking AI crawlers by default on new domains in July 2025, so a site added after that date may be blocking retrieval bots you never chose to block. A crawl of your site before and after any Cloudflare change, compared side by side, is the most reliable way to catch this. See how to run an SEO regression test after Cloudflare changes.
Frequently asked questions
- Does blocking AI crawlers in Cloudflare affect Google rankings?
- No, as long as Googlebot stays allowed. Blocking training crawlers and the Google-Extended token does not affect Google Search.
- Is AI Crawl Control free?
- Yes. Monitoring, crawler actions, and the managed robots.txt work on all plans. Free plan detection relies on user agent strings, and Markdown for Agents requires Pro or higher.
- Why is my most crawled path an image?
- Many AI crawlers, especially training bots, fetch images and media at high volume. Image heavy crawling usually costs bandwidth without producing referral traffic.
- What is the difference between allowed requests and HTTP 200 crawls?
- Allowed means Cloudflare let the request through. HTTP 200 means the page actually loaded. The difference is redirects, not modified responses, and errors such as 404s.
- What is the difference between Content Signals and blocking?
- Content Signals state preferences in robots.txt. Blocking in AI Crawl Control returns a 403 or 402 at the edge whether or not the crawler cooperates.
- Can I export full AI crawler logs?
- The dashboard shows aggregated metrics on all plans. Raw log export through Logpush requires an Enterprise plan. For page level detail on lower plans, filter the HTTP Traffic analytics by user agent and content type. See Cloudflare logs vs origin logs.
Key takeaway
Monitor in Cloudflare, but validate outside it. The dashboard shows volume, operators, and intent. The gap between allowed and successful requests, the live robots.txt, and real HTTP responses show what crawlers actually get, and those decide your visibility in AI search.
Related: Cloudflare for SEO · Diagnosing crawling problems on Cloudflare · Cloudflare logs vs origin logs