- Free tools
- Robots.txt Checker
FREE ROBOTS.TXT CHECKER AND TESTER
Robots.txt Checker: see what crawlers can actually reach
Validate robots.txt syntax and test any URL against its Allow and Disallow rules for Googlebot, Bingbot and AI crawlers like GPTBot and ClaudeBot. It does what the retired Search Console robots.txt tester did. No signup, and it runs in your browser.
- Instant crawl verdict
- No signup required
- Tests AI crawlers too
WHAT WE CHECK
Your robots.txt
robots.txt always lives at the root of the host, for example https://example.com/robots.txt.
Checked in your browser. Your robots.txt is never stored.
Results
Fetch or paste a robots.txt file on the left and click “Check robots.txt.” Results appear here.
HOW IT WORKS
From robots.txt file to a clear crawl verdict
Google retired the Search Console robots.txt tester in December 2023. Its replacement, the robots.txt report, shows fetch and parse errors but no longer tests a single URL. This tool fills that gap.
Load the file
Enter your domain to fetch the live robots.txt, or paste a draft before you deploy it. Testing a draft catches a stray
Disallow: /before it reaches production.Pick a crawler and URL
Choose Googlebot, Bingbot, an AI crawler or a custom user-agent token, then enter a full URL or a path with query parameters.
Read the verdict
See Allowed or Blocked, the exact rule and line that decided it, every syntax issue, and how each search and AI crawler treats the same path.
WHAT THE CHECKER LOOKS AT
The rules behind every crawl decision
The tester follows RFC 9309, the Robots Exclusion Protocol standard, and Google’s documented parsing behavior, so the verdict matches how Googlebot reads your file.
User-agent matching
A crawler obeys one group only: the most specific User-agent token that matches it, falling back to User-agent: *. Groups naming the same crawler are merged.
Longest match wins
Inside the group, the rule with the longest matching path decides. When Allow and Disallow are equally long, Allow wins, so Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/.
Wildcards and anchors
* matches any characters and $ anchors the end of the URL. Disallow: /*?sort= blocks sorted faceted URLs, Disallow: /*.pdf$ blocks PDF files only.
Syntax errors
Flags lines without a colon, rules placed before any User-agent, paths without a leading slash, misspellings like Dissallow, and relative Sitemap URLs.
Ignored directives
Notes directives Google skips: Noindex (unsupported since 2019), Crawl-delay (Bing and Yandex only), and the Yandex-only Host and Clean-param lines.
AI crawler access
Tests the same path for training crawlers like GPTBot, ClaudeBot and CCBot, AI search crawlers like OAI-SearchBot and PerplexityBot, and control tokens like Google-Extended.
ROBOTS.TXT CHECKER VS SEARCH CONSOLE
The report tells you it parsed. The tester tells you what it blocks.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results from links, so pair this check with a review of duplicate content and noindex tags across the site.
| What you get | Robots.txt Checker | Search Console robots.txt report |
|---|---|---|
| Test a specific URL | ✓ Any URL or path | No, retired in 2023 |
| Shows the deciding rule | ✓ Rule and line number | No |
| Crawlers covered | Google, Bing, AI crawlers, custom | Google only |
| Test drafts before deploying | ✓ Paste any version | Live file only |
| Works on any site | ✓ No verification | Verified properties only |
Other crawlers can interpret edge cases differently. Some bots ignore robots.txt entirely, so protect sensitive URLs with authentication.
FAQ
Robots.txt testing questions
What happened to the Google Search Console robots.txt tester?
Google retired the robots.txt tester in December 2023 and replaced it with the robots.txt report. The report shows which robots.txt files Google found, when they were crawled, and any fetch or parse errors, but it does not test whether a specific URL is allowed for a specific crawler.
Does a Disallow rule remove a page from Google?
No. robots.txt controls crawling, not indexing. A disallowed URL can still be indexed from links, usually without a description. To keep a page out of the index, allow crawling and add a noindex meta robots tag or X-Robots-Tag header.
How does Google decide between Allow and Disallow rules?
Google uses the group whose User-agent token matches the crawler most specifically, then applies the rule with the longest matching path. When an Allow and a Disallow rule are the same length, Allow wins.
What happens if robots.txt is missing or returns an error?
A 404 or other 4xx response means crawlers treat the site as having no restrictions. A 5xx server error makes Google pause crawling the site until the file can be fetched again, which can waste crawl budget.
How do I block AI training crawlers but stay in AI search?
Disallow training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended, and leave search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Each token is controlled separately, and the AI crawler table above shows the result for every one.
Is there a robots.txt size limit?
Yes. Google reads the first 500 KiB of a robots.txt file and ignores the rest. Google also caches robots.txt for up to 24 hours, so changes are not instant.
Test one URL here. Audit crawlability across every page with SiteAuditLint.
One rule is easy to check. Blocked pages, noindex conflicts and orphaned URLs across a whole site need a crawl.
Run Free Site Audit