Magento · Layered navigation, filter parameters, sort and limit URLs and robots.txt rules
Magento's layered navigation turns every attribute option into a link. By default each one is a plain anchor with a query parameter, so crawlers can follow every filter, every combination of filters, and every sort and page size option on top.
The goal is not to block filters entirely. Some facets, such as a brand or a material within a category, match real searches. The goal is to choose those deliberately and keep every other combination out of the index and, where possible, out of crawl paths.
Crawl with parameters→Count per parameter→Choose indexable facets→Apply rules→Recrawl and compare
What can go wrong
Each kind of parameter creates URLs at a different rate. Sort and limit options multiply everything else.
| Parameter | Example | What usually breaks |
|---|---|---|
| Attribute filter | ?color=49 | Every option of every filterable attribute becomes an indexable category variant |
| Combined filters | ?color=49&size=167 | Combinations grow multiplicatively, and parameter order can create two URLs for one result |
| Price range | ?price=20-30 | Ranges shift as prices change, creating URLs that later return empty pages |
| Sort order | ?product_list_order=price | Same products in a different order, multiplied by every filter URL |
| Page size and view mode | ?product_list_limit=36 ?product_list_mode=list | Exact duplicates of the category with a different layout |
| Pagination | ?p=4 | Combined with filters and sorts, deep pages of every variant become crawlable |
How one category grows
1 category/women/tops.html1 URL
+ 4 filtersColor, size, material and price, 10 options each40 URLs
+ combinationsTwo filters at a time across those options600+ URLs
+ sort, limit, pagesEvery URL above times sort orders, page sizes and pagesTens of thousands
Illustrative numbers. Run a scoped crawl on your largest category to get your own.
What to check
These numbers come from a crawl of one or two large categories with parameters followed.
URLsHow many URLs each parameter creates
IndexableHow many return 200 without noindex and with a self canonical
CanonicalWhere filter URLs point their canonical
Bot hitsHow often search bots request each pattern in server logs
| Check | Healthy result |
|---|---|
| Indexable filter URLs | Only the facets you chose, each with unique title and H1 |
| Sort, limit and mode URLs | Noindexed or canonical to the clean category, never in the sitemap |
| Combined filter URLs | Noindexed or canonical to the category, unless chosen |
| Parameter order | One URL per filter combination, or both orders canonical to one |
| Empty filter results | No crawlable links to filter combinations with zero products |
| robots.txt | Blocks only patterns that never need a noindex to be seen |
| Sitemap | No parameter URLs unless they are chosen indexable facets |
How to run the checks
- Pick your largest categories and crawl them with query parameters followed and a URL limit, so the crawl finishes and shows the shape of the problem.
- Group results by parameter name. Count total and indexable URLs for each filter, price, sort, limit, mode and page parameter.
- Inspect a few URLs per group. Check the robots meta and canonical in the cached HTML, since extensions often change both.
- Compare with server logs or Search Console crawl stats to see which patterns bots actually spend time on.
- List search demand for facets. Use Search Console queries and keyword research to find filter combinations people search for, such as a brand within a category.
- Save the crawl as the baseline before changing any rules.
How to identify the scope
The parameter that grows fastest tells you where to start.
Sort and limit dominateToolbar linksMost URLs carry product_list_order or product_list_limit: control the toolbar first.
Combinations dominateMulti-select filtersURLs with two or more filters: combined filters need noindex or no links.
One attribute dominatesFilterable attributeOne parameter with hundreds of options: review whether it should be filterable at all.
Filters indexable everywhereExtension or templateRobots and canonical wrong on every filter URL: a layered navigation extension or theme setting.
How to apply the fix
Chosen facet /women/tops/linen.html index, self canonical, unique H1
Other filters ?color=49 noindex,follow or canonical to /women/tops.html
Toolbar ?product_list_order= / _limit= / _mode= noindex, never linked as plain anchors
Never crawl /catalogsearch/ /catalog/product_compare/ Disallow in robots.txt
Other filters ?color=49 noindex,follow or canonical to /women/tops.html
Toolbar ?product_list_order= / _limit= / _mode= noindex, never linked as plain anchors
Never crawl /catalogsearch/ /catalog/product_compare/ Disallow in robots.txt
- Give chosen facets a real landing page. A clean URL through an SEO extension, a URL rewrite or a dedicated category, with its own title, H1 and intro copy.
- Noindex everything else consistently. Use noindex,follow on filter, sort, limit and mode URLs, or a canonical to the clean category. Pick one approach per pattern and apply it in one place.
- Reduce links to non-indexable variants. Many layered navigation extensions can render filters as form controls or remove anchor links for combined filters, which saves crawl budget that noindex alone does not.
- Use robots.txt carefully. A Disallow rule stops crawling but hides any noindex tag behind it. Reserve it for patterns that should never be crawled, such as internal search.
- Trim filterable attributes. Attributes with hundreds of options and no search demand can be removed from layered navigation or limited to the categories that need them.
How to validate the fix
- Deploy, flush the full page cache, and crawl the same categories with the same scope.
- Compare URL counts per parameter in Audit Comparison.
- Confirm indexable filter URLs exactly match your chosen facet list.
- Watch Search Console crawl stats and the indexed pages report over the following weeks.
- Schedule Scheduled SEO Audits after each release, since extension updates often reset these rules.
0Indexable sort, limit or mode URLs
100%Indexable facets on your chosen list
0Filter URLs in the sitemap outside that list
Downloadable checklist
Downloadable checklistMagento faceted navigation SEO checklist
0 of 16 done
Common mistakes
- Blocking all parameters in robots.txt and adding noindex too. Google cannot see a noindex on a URL it is not allowed to crawl, so blocked URLs can stay indexed from links.
- Canonicalizing every filter URL to the category while keeping them all linked. Canonicals are hints. Crawlers still spend time fetching every variant.
- Indexing filters with no unique content. A filtered grid with the category's title and H1 is a near duplicate.
- Forgetting sort and limit parameters. They multiply every other URL and never deserve indexing.
- Testing on a small category. The problem scales with catalog size. Audit the largest categories.
- Assuming an extension update kept your settings. Recrawl after every update to the layered navigation module.
Related resources
Choose the facets that earn indexing
Faceted navigation is a product feature first and an SEO risk second. Measure how many URLs each parameter creates, pick the few facets that match real searches, give them proper landing pages, and keep every other combination out of the index and away from crawlers.