Warning AEO / GEO
AI crawlers blocked in robots.txt
- Severity
- Warning
- Impact
- Medium
- Effort to fix
- Low
- Issue code
ai_crawlers_blocked
Why it matters
Some AI and answer-engine crawlers are disallowed, so the site may not be cited in AI search results.
How to fix it
Decide which AI crawlers you want to allow and update robots.txt accordingly.
What this means
The site's robots.txt blocks one or more known AI crawlers from the start URL. The evidence lists which ones. SiteAud!tLint checks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended.
Is this a problem?
It depends on what you want, because these crawlers do different jobs:
| Crawler | Purpose | Blocking it means |
|---|---|---|
| OAI-SearchBot, ChatGPT-User | ChatGPT search and fetching pages users ask about | You are not cited in ChatGPT answers |
| PerplexityBot | Perplexity's search index | You are not cited in Perplexity |
| Claude-SearchBot | Claude's web search | You are not cited in Claude's search answers |
| GPTBot, ClaudeBot, CCBot | Collecting content for model training | Your content is not used for training; no effect on search |
| Google-Extended, Applebot-Extended | Opt-out tokens for Gemini and Apple AI training | No effect on Google Search or Siri search |
Blocking training crawlers is a reasonable business decision. Blocking the search and retrieval crawlers removes you from AI answers and the referral traffic they send, which is usually not what a site intends.
Common causes
- A blanket rule copied from an article about blocking AI.
- A CDN or hosting setting that adds AI blocks to robots.txt automatically, such as Cloudflare's managed robots.txt.
- A
User-agent: *rule withDisallow: /.
How to fix it
- Decide per crawler, using the table above.
- Remove the
Disallowrules for crawlers you want to allow, or add specificAllow:groups for them. - If your CDN manages robots.txt, change the setting there, then check the live file at
/robots.txt.
Verify the fix
Recrawl. The evidence lists only the crawlers still blocked. See AI visibility to track whether you are cited.
How it affects your score
Finding this issue on any page costs the AEO / GEO score 3 points, plus up to 12 more depending on the share of crawled pages affected. Weakens rankings, crawling or measurement. Fix after critical issues.
Find the affected URLs
Open Crawl > Issues and pick AI crawlers blocked in robots.txt. Each affected URL is listed with its evidence, and Copy URLs or Export URLs hands the list to a developer. The issue also appears in the Action Plan, ranked by impact (medium) and effort (low), and in the issues CSV under the code ai_crawlers_blocked. After fixing, run a new audit and use Compare to confirm the URLs moved to fixed.