01 / Crawl
SiteAuditLint crawls websites from your own computer, not from a cloud server. Your crawl data, audit history and API keys stay on your machine, and you pay once instead of every month.
SiteAuditLint is a desktop SEO crawler. You enter a start URL, the async crawler follows internal links, reads robots.txt and XML sitemaps, and stores what it finds in a local database. Nothing is uploaded to a crawling service, and there is no subscription tied to crawl credits.
1. How local crawling works
The crawler requests HTML pages with its own user agent, SiteAuditLintBot/1.0, discovers links, and keeps going until it reaches the end of the site, the edition's page limit, or you press Stop. It records the status code, response time, redirect path, headers and HTML signals for each URL, then runs page checks and site-wide checks once the crawl finishes.
What happens during an audit
The crawl, the checks and the scores all run on your computer.
2. Automatic handling of HTTP 429 rate limits
Some hosts, notably Shopify, refuse requests that arrive too fast and answer with HTTP 429 Too Many Requests. Instead of reporting those pages as broken, the crawler adapts its crawl rate:
- All requests share one pace. On a 429 the whole crawl pauses for the server's Retry-After time, or 2, 4, 8 and 16 seconds.
- The pace slows to 0.5s, 1s, 2s and up to one request every 5 seconds for the rest of the crawl.
- Each page is retried up to 4 times.
- After 10 successful requests in a row the pace speeds up again by 25 percent.
WarningCould not audit: rate limited
The server kept answering "too many requests" (HTTP 429), even after SiteAud!tLint slowed down and retried, so this page was not checked. This is common on Shopify and some hosting providers and does not mean the page is broken.
How to fix: In Settings, set Parallel requests to 1 and Delay between requests to 1 second, then audit again.
A page that still returns 429 is excluded from broken link, sitemap and backlink checks, and the report shows an incomplete audit notice so the score is never based on pages that were not read.
3. Page limits by edition
| Edition | Pages per audit |
|---|---|
| Free (no key) | 100 |
| Starter | 1,000 |
| Professional | 10,000 |
| Agency | 100,000 |
| Unlimited | No limit |
Every edition is a one-time purchase with a license key. When the Free edition stops at 100 pages, the audit tells you how many more URLs were found but not checked, so you know exactly what the results leave out.
4. Where your audit data lives
All audits are stored in one SQLite file on your computer, which is what makes history, comparisons and before and after charts possible without an account.
Windows: %LOCALAPPDATA%\SiteAuditor\SiteAuditor\audits.db
macOS: ~/Library/Application Support/SiteAuditor/audits.db
Linux: ~/.local/share/SiteAuditor/audits.dbReplacing or moving the app folder never deletes this file, so updating to a new version keeps every saved audit.
5. Crawl settings you control
| Setting | What it does |
|---|---|
| Maximum pages | Caps the audit at your edition's limit or lower |
| Parallel requests | How many pages are fetched at the same time |
| Delay between requests | A fixed pause per request for sensitive hosts |
| Respect robots.txt | Skips URLs your robots.txt disallows |
| External link checking | Checks outbound links after the crawl; can be switched off |
| Render JavaScript | Optional Playwright rendering for JavaScript sites |
Frequently asked questions
Does SiteAuditLint crawl from a server?
No. Every request is sent from your computer. There is no server-side crawling, so audits of staging sites, intranets and sites behind an IP allowlist work as long as your machine can reach them.
How many pages can one audit crawl?
It depends on the edition: Free stops at 100 pages, Starter at 1,000, Professional at 10,000, Agency at 100,000 and Unlimited has no page limit.
Why does my Shopify audit show rate limited pages?
Shopify and some hosts answer fast crawlers with HTTP 429. SiteAuditLint slows down and retries automatically. Pages that still fail are reported as rate limited, never as broken. Setting 1 to 2 parallel requests with a 1 second delay gives the most complete Shopify audit.
Want the background first? Browse the SiteAuditLint Academy, or see all features.