SITEAUD!TLINT CONSULTING
Indexability and crawlability audit: get the right pages indexed
Investigate whether search engines can discover your important URLs, access their content, and receive consistent signals about which pages should be indexed. A consultant groups affected URLs, forms root-cause hypotheses, and gives you remediation priorities and validation steps.
Fixed quote after a short scoping call. No retainer required.
Founder · Technical SEO Consultant
Developer of SiteAud!tLint
Last reviewed
- Status
- 2,140 URLs reported as "Discovered, currently not indexed" in Search Console.
- Pattern
- 94% are paginated listing pages beyond page 5, linked only from the previous page.
- Hypothesis
- Deep pagination chains and a high volume of low-value URLs lower crawl priority for these pages.
- Fix
- Add category hub links to deeper listings and remove empty paginated pages from sitemaps.
- Validate
- Status count trend in the Page indexing report and crawl requests in logs.
Signs you need an indexability and crawlability audit
- Search Console shows many "Discovered" or "Crawled, currently not indexed" URLs.
- Important pages are excluded by noindex, robots.txt, or canonicals you did not intend.
- Sitemaps list URLs that redirect, error, or are noindexed.
- Crawlers get stuck in parameter or calendar URLs.
- Indexed page counts dropped after a deployment.
- You need to know exactly why specific URLs are not indexed.
What the indexability and crawlability audit covers
Robots.txt rules and crawl restrictions
Whether crawl rules block what should be crawled.
- Disallow rules on important pages or resources
- Rules that differ between environments
- Robots.txt blocking pages that also carry noindex
Meta robots and X-Robots-Tag
Whether indexing directives match your goals.
- Unintended noindex on templates
- X-Robots-Tag headers on files and pages
- Directives changed by plugins or releases
Canonical and indexability conflicts
Whether signals agree on the preferred URL.
- Canonicals to redirected, noindexed, or 4xx URLs
- Google-selected canonicals that differ from declared ones
- Conflicts between sitemaps, canonicals, and internal links
Status codes and redirects
Whether URLs return the right responses.
- Soft 404s and unexpected 4xx or 5xx
- Redirect chains and loops
- Temporary redirects used for permanent moves
XML sitemap consistency
Whether sitemaps list only indexable, canonical URLs.
- Non-200, noindexed, or canonicalized URLs in sitemaps
- Important URLs missing from sitemaps
- lastmod accuracy
Crawl traps, parameters, and orphans
URL patterns that waste crawling or hide pages.
- Infinite parameter and calendar combinations
- Session and tracking parameters
- Orphan pages with no internal links
Patterns we trace to a single cause
Noindex shipped to production
A staging noindex rule deployed live can deindex a whole site.
Canonical to a redirect
One template canonical pointing at a redirected URL affects every page using it.
Sitemap noise
Sitemaps full of redirects and errors weaken trust in the URLs you do want indexed.
Parameter crawl traps
Unbounded parameter combinations absorb crawling meant for real pages.
How the audit works
Scope the audit
We confirm goals, priority pages or sections, and recent changes that shape where to look first.
Collect evidence
We crawl the site and review Search Console, analytics, and other data you share.
Group by pattern
Issues are grouped by template, section, or URL pattern instead of listed one URL at a time.
Trace root causes
Each pattern is traced to the setting, template, or process that produces it.
Prioritize and validate
Fixes are ranked by impact, reach, and effort, and rechecked after release when validation is in scope.
What you receive
Crawl and indexability findings
Issues by Search Console status and URL pattern.
Affected URL groups
Every finding tied to the templates and patterns involved.
Root-cause hypotheses
Likely causes with the evidence for each and what would confirm it.
Remediation priorities
Fixes ranked by affected pages, importance, and effort.
Validation procedures
Steps to confirm each fix in Search Console, logs, and crawls.
What we need from you
- Google Search Console access (read-only is enough)
- XML sitemap locations
- Server logs (optional, recommended)
- Recent release or migration notes (optional)
Who this audit is for
A good fit if
- Important pages are not indexed
- Search Console statuses are unclear or growing
- Indexing dropped after a release
- You need a focused diagnosis, not a full technical audit
Included vs. optional
- Core: all six areas above
- Optional: log file analysis
- Separate: full technical SEO audit including rendering, speed, and schema
- Separate: implementation
Indexability and Crawlability Audit vs. other SEO services
Quoted to your scope.
Pick the engagement closest to what you need, tell us about your site, and you get a fixed quote after a short scoping call.
Questions, answered.
What does this audit cover?
URL discovery, crawl access, and indexation signals: robots.txt, meta robots, X-Robots-Tag, canonicals, status codes, redirects, sitemaps, crawl traps, and orphan pages.
How is it different from a technical SEO audit?
It focuses only on discovery, crawling, and indexation. A technical SEO audit also covers rendering, Core Web Vitals, structured data, hreflang, and HTTPS.
Why does Google not index my page?
Common causes include low perceived value, duplicate signals, weak internal links, crawl priority, or conflicting directives. The audit tests which apply to your URLs.
Can you guarantee indexing?
No. Google decides what to index. The audit removes technical barriers and conflicting signals within your control.
Do you need server logs?
They are optional but help confirm how search engines crawl the affected URLs.
How long does it take?
Most audits are delivered within one to two weeks.
Get a clear, prioritized plan
Send your website URL and what you are investigating, and we will scope an indexability and crawlability audit around your goals.