Rate limiting, HTTP 429 and Shopify
Why hosts like Shopify return 429 Too Many Requests, how SiteAud!tLint backs off and retries, and the crawl settings that give a complete audit.
Some hosts refuse requests that arrive too fast and answer with HTTP 429 "Too Many Requests". Shopify does this often, and so do some shared hosts, CDNs and bot protection services. SiteAud!tLint handles it automatically so a throttled page is never mistaken for a broken one.
What happens on a 429
- All requests share one pace. On a 429 the whole crawl pauses for the server's
Retry-Aftertime, or 2, 4, 8 then 16 seconds if the server gives none. - The pace slows for the rest of the crawl: 0.5 s, then 1 s, 2 s, up to one request every 5 seconds.
- The page is retried up to 4 times.
- After 10 successful requests in a row, the pace speeds up again by 25%, never faster than one request every 0.25 s on a host that has rate limited.
A page that still returns 429 after the retries is reported as Could not audit: rate limited, a warning. It is never counted as broken, and it is left out of broken link, sitemap and backlink checks. Status codes count it as "Rate limited".
Settings for Shopify and strict hosts
In Settings > Crawl Settings...:
| Setting | Value |
|---|---|
| Parallel requests | 1 or 2 |
| Delay between requests | 1 s |
This is slower but gives the most complete audit. The HTML report shows an Incomplete audit notice when pages were rate limited, so readers know the score does not cover every page.
Frequently asked questions
Does a 429 mean my site is broken?
No. It means the server limited how fast one visitor could request pages. Search engine crawlers have their own limits and are not affected by your SiteAud!tLint settings.
Can I whitelist the crawler instead?
If you control the firewall or CDN, allow the SiteAuditLintBot user agent and your IP address. Shopify's own limits cannot be changed, so use the slower settings.
Applies to SiteAud!tLint 0.3 and later. Current version 0.7.1.