Home›Checklists›Robots.txt Checklist

Free SEO checklist · 10 sections · 12 checks

Robots.txt Checklist

A robots.txt checklist verifies that your robots.txt file is accessible, valid, and allows search engines and the AI crawlers you choose to reach important content, while blocking only what you intend.

12 Checks7 High priority10 SectionsPDF · Word · Excel Formats
Download checklist

SiteAuditLint checklist library

What this checklist covers

A robots.txt checklist verifies that your robots.txt file is accessible, valid, and allows search engines and the AI crawlers you choose to reach important content, while blocking only what you intend.

It is organized into 10 sections: File Accessibility, Syntax, User-agent Rules, Allow and Disallow, Search Crawlers, AI Crawlers, Sitemap Declaration, Wildcards, Testing and Deployment. Each check lists exactly what to verify and a priority, so you can work through the highest impact items first.

For background on the concepts behind these checks, see Robots.txt lesson, Check crawlability for Google, Bing and AI bots, Robots.txt and AI crawlers guide.

Who it is for:

Developerstechnical SEOs

Download the robots.txt checklist

Use the interactive version below, or download it to share with your team, attach to tickets, or work through offline.

Robots.txt Checklist

Work through each check

0 of 12 checks complete

File Accessibility

Syntax

User-agent Rules

Allow and Disallow

Search Crawlers

AI Crawlers

Sitemap Declaration

Wildcards

Testing

Deployment

Progress is saved in this browser only.

Detailed explanations

Each section below explains what to check, why it matters, the problems you will usually find, how to fix them, and how to confirm the fix.

File Accessibility

What to check:

  • Check file location. File is at the root and returns 200.
  • Check file size. File is under 500 KiB.

Why it matters: Robots.txt controls which paths crawlers may request. It is the fastest way to block an entire site by accident. It also governs access for AI crawlers, so it is now part of your AI search visibility strategy as well as your traditional SEO.

Common problems:

  • A staging Disallow: / rule deployed to production
  • CSS and JavaScript blocked, stopping Google from rendering pages
  • Overly broad wildcard patterns blocking important directories
  • Missing or outdated sitemap declaration

How to fix it: Keep robots.txt minimal, block only what you truly need to keep out of crawlers, allow rendering resources, and declare your XML sitemap with an absolute URL.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

Learn more: Robots.txt lesson, Robots.txt and AI crawlers guide, Blocked by robots.txt issue, Missing robots.txt issue.

How SiteAuditLint helps: it checks file accessibility across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Syntax

What to check:

  • Validate syntax. Directives parse correctly with no typos.

Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

How SiteAuditLint helps: it checks syntax across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

User-agent Rules

What to check:

  • Check user-agent groups. Groups are correctly scoped.

Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

How SiteAuditLint helps: it checks user-agent rules across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Allow and Disallow

What to check:

  • Check disallow rules. No important paths blocked.
  • Check CSS and JS access. Rendering resources are not blocked.

Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

How SiteAuditLint helps: it checks allow and disallow across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Search Crawlers

On this checklist: check Googlebot and Bingbot. Explained in full in the Technical SEO Checklist.

AI Crawlers

What to check:

  • Check AI crawler rules. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot rules match policy.

Why it matters: Search engines can only rank what they can discover. See the earlier section on this topic for common problems.

How to verify: Run a full crawl, compare discovered URLs with your sitemap and analytics landing pages, and review the crawl depth report for any important URL deeper than three clicks.

How SiteAuditLint helps: it checks ai crawlers across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Sitemap Declaration

On this checklist: declare sitemap. Explained in full in the Technical SEO Checklist.

Wildcards

What to check:

  • Test wildcards. * and $ patterns match only intended URLs.

Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

How SiteAuditLint helps: it checks wildcards across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Testing

What to check:

  • Test key URLs. Key URLs tested against live rules.

Why it matters: Robots.txt controls which paths crawlers may request. See the earlier section on this topic for common problems.

How to verify: Fetch /robots.txt on production, test a list of key URLs against the live rules for Googlebot and each AI user agent, and confirm the file returns 200.

How SiteAuditLint helps: it checks testing across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Deployment

What to check:

  • Check after deploy. Production file is not the staging file.

Why it matters: Launches often ship staging settings to production. A single forgotten noindex or Disallow can keep a new site out of search for weeks.

Common problems:

  • Staging noindex left on production
  • Robots.txt still disallowing everything
  • Canonicals pointing to the staging domain
  • Analytics not installed

How to fix it: Use a launch runbook that removes staging protections, swaps domains in canonicals and sitemaps, and verifies tracking.

How to verify: Crawl production immediately after launch and compare against the final staging crawl.

Learn more: Pre-launch website checklist, Run your first audit, Noindex issues.

How SiteAuditLint helps: it checks deployment across every crawled URL instead of one page at a time and lists exactly which URLs are affected.

Common mistakes

Shipping a staging Disallow: / to production

Blocking CSS and JavaScript

Using robots.txt to remove pages from the index

Copying AI crawler blocks without a policy decision

Forgetting that rules are case sensitive

When to run the checklist

  • Before and after every deployment
  • Before launching a site
  • When setting an AI crawler policy
  • When pages unexpectedly drop from search

How to verify fixes

Save a baseline crawl before you change anything. Fix high-priority and template-level issues first, since one template fix resolves many URLs, then re-crawl with the same start URL and settings and compare the two audits with audit comparison. Finally, confirm important pages in Search Console with URL Inspection. See how to prioritize audit findings.

SiteAuditLint workflow

Most checks in this list run automatically in a SiteAuditLint crawl. Use audit comparison to diff crawls and scheduled audits to monitor for regressions.

All SEO checklists

See the full checklist library, with a guide to choosing the right checklist for each job.