SiteAud!tLint Support

Opportunity Technical SEO

No robots.txt file

Severity
Opportunity
Impact
Low
Effort to fix
Low
Issue code
robots_missing

Why it matters

robots.txt returned an error. Crawlers fall back to crawling everything and you lose a place to declare sitemaps.

How to fix it

Publish a robots.txt at the site root with a Sitemap: line.

What this means

The site has no robots.txt: the request for /robots.txt returned an error status or no response. The evidence shows what came back.

Why it matters

A missing robots.txt (a 404) is not an error for search engines: they treat it as "everything may be crawled". This is an opportunity, because a robots.txt file is the standard place to:

  • Declare your XML sitemap, so every search engine finds it without a manual submission.
  • Keep crawlers out of low-value areas, such as internal search results, cart and checkout pages, and endless filter combinations.
  • Set rules for AI crawlers, if you want to allow or block specific ones. See AI crawlers blocked in robots.txt.

When it is a real problem

If robots.txt returns a 5xx error or times out, Google treats the whole site as temporarily blocked and pauses crawling until it gets a valid answer. Fix that immediately.

A minimal robots.txt

User-agent: *
Disallow: /cart
Disallow: /search

Sitemap: https://example.com/sitemap.xml

How to add one

  1. Create a plain text file named robots.txt at the root of the domain, so it loads at https://example.com/robots.txt. Each subdomain needs its own.
  2. Start with the sitemap line, and only add Disallow rules for areas you are sure should not be crawled.
  3. Never use Disallow: / on a live site; it blocks everything.

Many platforms generate robots.txt for you and let you edit it in their settings.

Verify the fix

Recrawl. The issue clears once robots.txt returns a 200. Check Blocked by robots.txt afterwards to make sure no important page is blocked by a new rule.

How it affects your score

Finding this issue on any page costs the Technical SEO score 0.5 points, plus up to 4 more depending on the share of crawled pages affected. A smaller improvement with a modest effect on its own.

Find the affected URLs

Open Crawl > Issues and pick No robots.txt file. Each affected URL is listed with its evidence, and Copy URLs or Export URLs hands the list to a developer. The issue also appears in the Action Plan, ranked by impact (low) and effort (low), and in the issues CSV under the code robots_missing. After fixing, run a new audit and use Compare to confirm the URLs moved to fixed.

Guides that cover this issue