- Free tools
- Robots.txt Generator
FREE ROBOTS.TXT GENERATOR
Robots.txt Generator: control what crawlers see
A free robots.txt generator. Set your allow and disallow rules, block AI crawlers with one click, add your sitemap, then copy or download a valid robots.txt. No signup, nothing leaves your browser.
- One-click AI crawler blocks
- Presets + sitemap
- No signup required
CRAWLERS YOU CAN CONTROL
Build your rules
BLOCK AI CRAWLERS
Tick the AI and answer-engine bots you want to disallow. Each adds a full block for that user-agent.
robots.txt
Generated in your browser. Nothing is uploaded. Place the file at your site root: /robots.txt
HOW IT WORKS
From crawl rules to a valid robots.txt in seconds
robots.txt sits at your site root and tells crawlers which paths they may fetch. Get a rule wrong and you can hide pages from Google by accident, or leave private areas wide open.
Choose your rules
Start from a preset or add your own Disallow and Allow paths. The file updates live as you type.
Block or allow crawlers
Tick the AI crawlers you want to keep out, and add your sitemap so search engines find your URLs.
Save as robots.txt
Copy or download the file and upload it to your site root at /robots.txt, then confirm it with the Robots.txt Checker.
WHAT THIS TOOL GENERATES
Every directive a robots.txt needs
Clean, valid syntax for the directives crawlers actually read, with the AI-crawler blocks built in.
Per-crawler rules
A default block for all crawlers plus dedicated blocks for any bots you want to treat differently.
Disallow & Allow
List the paths to block, and carve out exceptions with Allow, each on its own line in the right order.
AI crawler blocks
One-click Disallow for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more, to control AI training and answer engines.
Crawl-delay
Add an optional Crawl-delay to ask slower-crawling bots to space out their requests on heavy sites.
Sitemap directive
Point crawlers at one or more XML sitemaps so they discover and prioritise your URLs.
Ready-made presets
Start from Allow everything, Block everything, WordPress or Block AI crawlers, then fine-tune.
GENERATOR VS HAND-CODING
One wrong line can deindex your whole site
A stray Disallow: / blocks everything; the wrong order of Allow and Disallow has no effect. A generator writes valid, predictable rules, then you confirm them with the Robots.txt Checker.
| What you get | This Generator | Hand-coding | No robots.txt |
|---|---|---|---|
| Valid syntax | ✓ Always | Error-prone | - |
| One-click AI crawler blocks | ✓ 10 bots | Look up each | All allowed |
| Sitemap directive | ✓ | Often forgotten | No |
| Risk of blocking the whole site | Low, previewed | High | - |
| No coding needed | ✓ | Manual | ✓ |
| Verify & crawl the whole site | ✓ With SiteAuditLint | Manual | No |
robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex tag, not Disallow - a disallowed URL can still be indexed from links.
FAQ
Robots.txt generator questions
What is a robots.txt generator?
A robots.txt generator turns your crawl rules into a valid robots.txt file. You set which paths to allow or disallow and which crawlers to block, and the tool writes the file for you to copy or download.
Where do I put the robots.txt file?
At the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. Crawlers only read it there; a robots.txt in a subfolder is ignored.
How do I block AI crawlers like GPTBot?
Tick GPTBot, ClaudeBot, PerplexityBot and the others in the AI crawlers section. Each adds a block that disallows that user-agent from your whole site. The bots must choose to respect it, and the major ones do.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in search if other pages link to it. To keep a page out of the index, allow crawling and add a noindex meta tag instead.
What does Crawl-delay do?
It asks a crawler to wait a number of seconds between requests. Googlebot ignores it, but Bing and some other bots honour it, which can help on servers under load.
Should I add my sitemap to robots.txt?
Yes. A Sitemap line helps search engines find your XML sitemap even if you have not submitted it in their webmaster tools. You can list more than one.
Generate robots.txt here. Audit crawlability across every page with SiteAuditLint.
A single Disallow rule can quietly block hundreds of pages. SiteAuditLint crawls your whole site and flags every URL blocked by robots.txt.
Download SiteAuditLint