AI Crawlers vs Traditional Search Crawlers

AI OVERVIEW

Blocking an AI training bot doesn't block that company's search bot or user fetcher. Each needs its own robots.txt rule.

For twenty years, "crawler" mostly meant Googlebot and Bingbot. Now your server logs are full of GPTBot, ClaudeBot, PerplexityBot, and a dozen other user agents, and each one wants something different from your pages.

The biggest misunderstanding is treating "AI crawlers" as one thing. Most AI companies run several bots with different jobs, and a robots.txt rule for one does not touch the others. This guide explains how AI crawlers differ from traditional search engine crawlers, how to control each one in robots.txt, how to verify who is really crawling, and what your server logs reveal.

Last reviewed: [add date]. Bot names, user agent tokens, and crawler policies change often, so check each operator's documentation before changing your robots.txt.

How traditional search crawlers work

Googlebot and Bingbot do one main job: fetch pages so they can be indexed, ranked, and shown in search results.

01They follow robots.txtRules are well documented and consistently honored.
02They render JavaScriptGooglebot uses an evergreen version of Chromium to render pages.
03They adapt crawl rateCrawl speed adjusts to how quickly your server responds.
04They can be verifiedOperators publish reverse DNS hostnames and IP ranges to confirm a request is genuine.

Blocking them takes you out of their search results. That is why a stray Disallow: / for Googlebot is one of the most damaging mistakes in technical SEO.

AI crawlers come in three types

The group a bot belongs to decides what blocking it actually does.

Type 1Training crawlersCollect public content that may train future models. Blocking affects future training only, not AI answers today.
Type 2Search index crawlersBuild the index an AI product searches when answering. Block these and you are less likely to be found and cited.
Type 3User-triggered fetchersVisit a page in real time because a user asked. Some operators treat them differently under robots.txt.
OperatorTrainingSearch indexUser-triggered
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
Perplexity(none listed)PerplexityBotPerplexity-User
One AI company Training botDisallow: / Search botstill allowed User fetcherstill allowed Each name is a separate robots.txt token with its own rule.
Anthropic, for example, documents all three bots separately. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.

User-triggered fetchers are where policies differ most. Anthropic states that all its bots, including Claude-User, honor robots.txt. OpenAI and Perplexity have said user-initiated fetches may not follow robots.txt the same way automated crawling does. If keeping content out of AI tools completely matters, robots.txt alone may not be enough.

Block a bot, not a company.

Tokens that are not crawlers

TokenGoogle-ExtendedControls whether content fetched by Google's normal crawlers can be used for Gemini training and grounding. It does not affect Google Search. AI Overviews are part of Search, so blocking Google-Extended does not remove you from them.
TokenApplebot-ExtendedWorks the same way for Apple's AI training. Applebot itself still crawls for Siri and Spotlight.
CrawlerCCBot and othersCommon Crawl's public dataset is widely used for training, and bots from Meta, ByteDance, Amazon, and others also visit. Not all document their behavior clearly.

What actually differs between AI and search crawlers

Search crawlersAI crawlers
Main purposeBuild a search indexModel training, an AI search index, or live fetches for a user
Bots per operatorMostly one main botOften three, each controlled separately
JavaScriptGooglebot and Bingbot render itMany appear to read only raw HTML
Effect of blockingRemoved from search resultsDepends entirely on which bot you block
VerificationReverse DNS and published IP rangesPublished IP ranges from some operators, inconsistent across others
Crawl-delayIgnored by Googlebot, respected by BingSupported by some (Anthropic documents it), not all

The JavaScript row matters most. If main content only appears after client-side rendering, a crawler that reads raw HTML sees an empty shell. Content you want AI search tools to find and cite belongs in the HTML your server sends, through server-side rendering or static generation. Our JavaScript SEO guide covers raw vs. rendered HTML.

How to write robots.txt for AI crawlers

Each bot follows the most specific user-agent group that names it. A common setup allows AI search and live fetches but opts out of training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

01A named group replaces the * group

Once you add User-agent: GPTBot, rules under User-agent: * such as Disallow: /admin/ no longer apply to GPTBot. Repeat them in each named group.

02robots.txt is a request, not a lock

Well-behaved bots follow it. Scrapers pretending to be GPTBot will not. Use authentication or server-level blocking for private content.

03Blocking by IP can backfire

Anthropic notes that blocking its IPs may stop its bots from reading your robots.txt at all, so an IP-based opt-out may not work reliably.

04Old tokens linger

Many files still list deprecated names like anthropic-ai or Claude-Web. Harmless, but they do not control current bots. Test your rules with the free robots.txt checker.

How to verify who is really crawling

User-agent strings are trivial to spoof. Anyone can send User-Agent: Googlebot. Verify the important ones before acting on your logs:

Request IPFrom your access log
Reverse DNSgooglebot.com, google.com, or search.msn.com
Forward DNSHostname returns the same IP
VerifiedOr match published IP ranges
BotHow to verifyReliability
GooglebotReverse DNS resolves to googlebot.com or google.com, then forward lookup returns the same IP. Google also publishes crawler IP ranges as JSON.Strong
BingbotReverse DNS resolves to search.msn.com. Bing Webmaster Tools also has a verification tool.Strong
AI crawlersSome operators, including OpenAI, publish IP ranges. A "GPTBot" request from an IP not on OpenAI's list is not GPTBot.Varies

What your server logs can tell you

A site crawl shows what bots could access. Server logs show what they actually requested. You need both.

  1. Which bots visit at all. Many sites find AI bots visit far more often than expected, or not at all.
  2. Status codes per bot. Mostly 403s or 429s means something is blocking it, often a firewall, WAF, or CDN bot protection rule.
  3. What they request. Important pages, or mostly old URLs, parameters, and 404s?
  4. Whether they obey robots.txt. Requests from a verified bot to a disallowed path are worth reporting to the operator.
  5. Load. A Crawl-delay (for bots that support it) or temporary 429 and 503 responses can slow a heavy bot. Google advises against returning those to Googlebot for long periods.

Logs usually live with your host, CDN, or server, not in a crawler, so this part needs access to those systems.

A crawl shows what bots could see. Logs show what they did.

A quick AI crawler audit checklist

1Read robots.txt per botWhich group applies to each user agent, and what it allows.
2Confirm blocks are intentionalLeftover Disallow rules, deprecated tokens, named groups that dropped your * rules.
3Check CDN and firewall settingsBot protection can block AI search crawlers whatever robots.txt says.
4Confirm content is in raw HTMLMain content, links, and canonical tags present before JavaScript runs.
5Review a week of logsBy verified bot, status code, and URL type.
6Recheck after every changeEach robots.txt edit, CDN rule, or new bot in an operator's lineup.

Checking AI crawler access with SiteAuditLint

The SiteAuditLint AEO / GEO audit checks nine major AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, against your robots.txt and shows which are allowed or blocked. It also detects llms.txt, a proposed file for guiding AI tools that major search engines have not confirmed as a ranking signal, and flags content structure issues. It all rolls up into an Answer Engine Readiness score.

Access is only half the question. The AI visibility tracker runs your own prompts through ChatGPT, Gemini, Claude, and Perplexity and reports how often your pages are cited, so you can see whether allowing those crawlers is paying off.

On our own site (714 URLs, audited November 13, 2026), all nine AI crawler checks passed and Answer Engine Readiness scored 98 out of 100. The audit still flagged 243 pages with no question-led sections, a reminder that access alone does not make content easy for answer engines to extract and cite.

SiteAuditLint AEO and GEO view showing Answer Engine Readiness 98 out of 100, 9 of 9 AI crawlers allowed including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended, plus llms.txt detected
AEO / GEO view: AI crawler access, llms.txt detection, and answer-ready content per URL.

Frequently asked questions

What is the difference between AI crawlers and search engine crawlers?

Search engine crawlers like Googlebot and Bingbot fetch pages to build a search index. AI crawlers may collect content for model training, build an AI search index, or fetch a page live because a user asked, and one AI company often runs a separate bot for each job.

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, and live fetches come from ChatGPT-User, so each needs its own robots.txt rule.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether your content is used for Gemini training and grounding. AI Overviews are part of Google Search, which is governed by Googlebot.

Do AI crawlers render JavaScript?

Many appear to read only raw HTML. If your main content appears only after client-side rendering, AI search tools may see an empty page, so serve important content in the initial HTML.

How do I know if a bot claiming to be GPTBot is real?

Check the request IP against the IP ranges OpenAI publishes. User-agent strings are easy to fake, so a request from an IP outside the published list is not GPTBot.

Is robots.txt enough to keep content out of AI tools?

Not always. Well-behaved bots follow it, but some user-triggered fetchers may not apply it the same way, and scrapers ignore it. Use authentication or server-level blocking for content that must stay private.

Decide. Write. Verify. Serve.

Decide per bot, write each rule explicitly, verify who is really visiting, and keep the content you want found in the raw HTML.

AI search does not replace traditional search. It adds a second set of crawlers with different rules, and your robots.txt now has to speak to both.

One company, three bots

Download SiteAuditLint free