SEO QA · Verification process
HTTP header SEO testing checks the response headers search engines read before they look at any HTML. A noindex in the X-Robots-Tag header, a canonical in a Link header, or a 503 without Retry-After all change how a page is treated, yet none of them are visible when you view the page in a browser.
Definition
Header testing requests URLs, records the full response headers, and compares them with expected values per template. Because headers can be added or rewritten by the application, the web server, the CDN and the firewall, a header test is also a test of which layer is responsible.
Headers that affect SEO
| Header | What it controls | Expected on indexable URLs | Common defect |
|---|---|---|---|
| Status line | Whether the URL is live, moved or gone | 200 | Soft errors, wrong 3xx type |
| X-Robots-Tag | Indexing and snippet directives | Absent, or no noindex/none | noindex added by CDN or middleware |
| Link: rel=canonical | Canonical, often on PDFs and files | Matches HTML canonical or absent | Conflicts with HTML canonical |
| Link: rel=alternate hreflang | Language alternates for non-HTML files | Matches the hreflang set | Stale alternates after URL changes |
| Location | Redirect target | Final URL, absolute | Points to another redirect |
| Content-Type | How the body is parsed | text/html; charset=utf-8 | Wrong type or charset |
| Retry-After | Signals temporary unavailability with 503 | Present during maintenance | 503 without it, or 200 maintenance pages |
| Strict-Transport-Security | Forces HTTPS | Present | Missing, see HSTS missing |
| Vary | Which request headers change the response | Accurate for user agent or language | Missing when content differs by user agent |
Which layer set the header?
| Layer | Typical change | How to isolate it |
|---|---|---|
| Application | Middleware adds headers per route | Request the origin directly, bypassing the CDN |
| Web server | Config rules per directory or file type | Compare server config between environments |
| CDN / edge | Header rules, cached responses | Compare a cache hit with a cache miss |
| WAF / bot protection | Challenge pages or blocks for crawlers | Compare a crawler user agent with a browser user agent |
The last row is the most common surprise: the page works for people and fails for crawlers. Compare responses for both user agents on the same URL. Blocks aimed at AI crawlers are covered in fixing 403 errors for AI crawlers.
Header and HTML conflicts
| Conflict | Effect | Fix |
|---|---|---|
| HTML index, header noindex | Header wins, page excluded | Remove the header rule |
| HTML canonical A, Link header canonical B | Conflicting signals, both may be ignored | Keep one source |
| 200 status on a maintenance page | Maintenance text may get indexed | Return 503 with Retry-After |
| Redirect via meta refresh while header is 200 | Weaker, slower redirect | Use a 301 or 308 header |
When to test
- CDN, cache or firewall rule changes
- Server or hosting configuration changes
- New middleware or framework upgrades
- Planned maintenance windows
- Changes to how PDFs and downloadable files are served
Before and after example
HTTP 200
X-Robots-Tag: (none)
HTTP 200
X-Robots-Tag: noindex
The page looked unchanged in every browser. Only the header comparison exposed it, which is why post-deployment testing should always include headers.
Acceptance criterion
QA test: Crawl all templates, record response headers with a crawler user agent.
Expected: No noindex or none in X-Robots-Tag on URLs listed in the sitemap.
Failure: Any blocking value on an indexable URL.
Verification: Repeat after every CDN, server or firewall change.