SiteAuditLint checklist library
What this checklist covers
A robots.txt checklist verifies that your robots.txt file is accessible, valid, and allows search engines and the AI crawlers you choose to reach important content, while blocking only what you intend.
It is organized into 10 sections: File Accessibility, Syntax, User-agent Rules, Allow and Disallow, Search Crawlers, AI Crawlers, Sitemap Declaration, Wildcards, Testing and Deployment. Each check lists exactly what to verify and a priority, so you can work through the highest impact items first.
For background on the concepts behind these checks, see Robots.txt lesson, Check crawlability for Google, Bing and AI bots, Robots.txt and AI crawlers guide.
Who it is for:
Download the robots.txt checklist
Use the interactive version below, or download it to share with your team, attach to tickets, or work through offline.
Robots.txt Checklist
Work through each check
0 of 12 checks complete
File Accessibility
Syntax
User-agent Rules
Allow and Disallow
Search Crawlers
AI Crawlers
Sitemap Declaration
Wildcards
Testing
Deployment
Progress is saved in this browser only.
Detailed explanations
Each section below explains what to check, why it matters, the problems you will usually find, how to fix them, and how to confirm the fix.
File Accessibility
What to check:
- Check file location. File is at the root and returns 200.
- Check file size. File is under 500 KiB.
Why it matters: Robots.txt controls which paths crawlers may request. It is the fastest way to block an entire site by accident. It also governs access for AI crawlers, so it is now part of your AI search visibility strategy as well as your traditional SEO.
Common problems:
- A staging Disallow: / rule deployed to production
- CSS and JavaScript blocked, stopping Google from rendering pages
- Overly broad wildcard patterns blocking important directories
- Missing or outdated sitemap declaration
How to fix it: Keep robots.txt minimal, block only what you truly need to keep out of crawlers, allow rendering resources, and declare your XML sitemap with an absolute URL.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
Learn more: Robots.txt lesson, Robots.txt and AI crawlers guide, Blocked by robots.txt issue, Missing robots.txt issue.
How SiteAuditLint helps: it checks file accessibility across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Syntax
What to check:
- Validate syntax. Directives parse correctly with no typos.
Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
How SiteAuditLint helps: it checks syntax across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
User-agent Rules
What to check:
- Check user-agent groups. Groups are correctly scoped.
Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
How SiteAuditLint helps: it checks user-agent rules across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Allow and Disallow
What to check:
- Check disallow rules. No important paths blocked.
- Check CSS and JS access. Rendering resources are not blocked.
Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
How SiteAuditLint helps: it checks allow and disallow across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Search Crawlers
On this checklist: check Googlebot and Bingbot. Explained in full in the Technical SEO Checklist.
AI Crawlers
What to check:
- Check AI crawler rules. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot rules match policy.
Why it matters: Search engines can only rank what they can discover. See the earlier section on this topic for common problems.
How to verify: Run a full crawl, compare discovered URLs with your sitemap and analytics landing pages, and review the crawl depth report for any important URL deeper than three clicks.
How SiteAuditLint helps: it checks ai crawlers across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Sitemap Declaration
On this checklist: declare sitemap. Explained in full in the Technical SEO Checklist.
Wildcards
What to check:
- Test wildcards. * and $ patterns match only intended URLs.
Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
How SiteAuditLint helps: it checks wildcards across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Testing
What to check:
- Test key URLs. Key URLs tested against live rules.
Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.
How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.
How SiteAuditLint helps: it checks testing across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Deployment
What to check:
- Check after deploy. Production file is not the staging file.
Why it matters: Launches often ship staging settings to production. A single forgotten noindex or Disallow can keep a new site out of search for weeks.
Common problems:
- Staging noindex left on production
- Robots.txt still disallowing everything
- Canonicals pointing to the staging domain
- Analytics not installed
How to fix it: Use a launch runbook that removes staging protections, swaps domains in canonicals and sitemaps, and verifies tracking.
How to verify: Crawl production immediately after launch and compare against the final staging crawl.
Learn more: Pre-launch website checklist, Run your first audit, Noindex issues.
How SiteAuditLint helps: it checks deployment across every crawled URL instead of one page at a time and lists exactly which URLs are affected.
Common mistakes
Shipping a staging Disallow: / to production
Blocking CSS and JavaScript
Using robots.txt to remove pages from the index
Copying AI crawler blocks without a policy decision
Forgetting that rules are case sensitive
When to run the checklist
- Before and after every deployment
- Before launching a site
- When setting an AI crawler policy
- When pages unexpectedly drop from search
How to verify fixes
Save a baseline crawl before you change anything. Fix high-priority and template-level issues first, since one template fix resolves many URLs, then re-crawl with the same start URL and settings and compare the two audits with audit comparison. Finally, confirm important pages in Search Console with URL Inspection. See how to prioritize audit findings.
SiteAuditLint workflow
Most checks in this list run automatically in a SiteAuditLint crawl. Use audit comparison to diff crawls and scheduled audits to monitor for regressions.
All SEO checklists
See the full checklist library, with a guide to choosing the right checklist for each job.