Warning Technical SEO
Blocked by robots.txt
- Severity
- Warning
- Impact
- Medium
- Effort to fix
- Low
- Issue code
blocked_robots
Why it matters
Crawlers are disallowed from fetching this URL, so its content cannot be evaluated or ranked on its merits.
How to fix it
If this page should rank, remove or narrow the matching Disallow rule in robots.txt.
What this means
The site's robots.txt disallows this URL for the crawler, so SiteAud!tLint did not fetch it. The rules are read for the crawler's user agent, and for the * group that applies to every crawler.
Because the page was not fetched, no other checks run on it.
Why it matters
robots.txt controls crawling, not indexing. A blocked page:
- Cannot be read, so search engines see no content, title, links or noindex on it.
- Can still be indexed if other pages link to it, often shown with no description.
- Cannot pass its signals onward, because its links and canonical tag are never seen.
That makes robots.txt the right tool for keeping crawlers out of areas like internal search results, carts and endless filter combinations, and the wrong tool for removing pages from search.
Check that the block is intentional
| Blocked URL | Usually |
|---|---|
| Admin, cart, checkout, account, internal search | Intended |
| Filter and sort parameters on large catalogues | Intended, if the clean category pages are open |
| Product, article or category pages | A mistake: a rule is too broad |
| CSS or JavaScript files | A mistake: Google needs them to render pages |
| Everything | A staging Disallow: / copied to the live site |
How to fix a block that should not be there
- Open
/robots.txtand find the rule that matches the URL. Rules match from the start of the path, soDisallow: /palso blocks/productsand/pricing. - Narrow the rule, or add a more specific
Allow:line. - To remove a page from search, allow crawling and use a noindex instead, so search engines can see it.
Verify the fix
Recrawl. Unblocked URLs are fetched and checked like any other page. Leave Respect robots.txt on in Crawl settings to see what crawlers see.
How it affects your score
Finding this issue on any page costs the Technical SEO score 3 points, plus up to 12 more depending on the share of crawled pages affected. Weakens rankings, crawling or measurement. Fix after critical issues.
Find the affected URLs
Open Crawl > Issues and pick Blocked by robots.txt. Each affected URL is listed with its evidence, and Copy URLs or Export URLs hands the list to a developer. The issue also appears in the Action Plan, ranked by impact (medium) and effort (low), and in the issues CSV under the code blocked_robots. After fixing, run a new audit and use Compare to confirm the URLs moved to fixed.