01 / Desktop crawler
Screaming Frog SEO Spider is a desktop website crawler used by technical SEOs to collect detailed data about every URL on a site. It runs on your own computer, crawls like a search engine bot, and exports raw data for analysis, which makes it one of the most widely used tools for technical SEO audits.
This lesson explains how the SEO Spider works, its key features, how it fits into the audit process, and where its strengths and limits lie. For background on crawlers in general, see what a website crawler is.
1. What Screaming Frog is
The SEO Spider is a desktop application for Windows, macOS, and Linux. It starts from a URL, follows links, and records status codes, titles, meta descriptions, headings, canonicals, directives, response times, and more. A free version crawls a limited number of URLs, and a paid annual licence removes the limit and unlocks advanced features.
SEO tools update plans, limits, and features frequently. This lesson focuses on how each tool works and where it fits in an audit workflow. Check the vendor's website for current pricing and plan limits before choosing.
2. Key features
| Feature | What it does |
|---|---|
| Spider and list mode | Crawl a site from a start URL, or audit a supplied list of URLs such as a migration map |
| JavaScript rendering | Render pages with a headless Chromium browser and compare raw and rendered HTML |
| Custom extraction | Pull any element with XPath, CSS selectors, or regex, such as prices or schema fields |
| Custom search | Find pages containing, or missing, specific text or code |
| API integrations | Connect Google Search Console, Google Analytics, PageSpeed Insights, and backlink tools |
| Crawl comparison | Compare two crawls to see new, removed, and changed issues |
| Database storage mode | Store crawl data on disk to handle very large sites |
| Scheduling and CLI | Run crawls on a schedule or from the command line |
3. A typical Screaming Frog audit workflow
- Configure the crawlSet user agent, rendering, speed, include and exclude rules, and robots.txt behaviour.
- Connect data sourcesAdd Search Console, Analytics, and PageSpeed data to each URL.
- Crawl the siteMonitor memory and response codes during the crawl.
- Run crawl analysisCalculate link metrics, orphan pages from sitemaps, and duplicate content.
- Review tabs and filtersWork through Response Codes, Page Titles, Directives, Canonicals, and other tabs.
- Export dataExport to spreadsheets or reports for prioritization and findings.
4. Strengths
Depth and flexibility
Almost every crawl setting can be adjusted, and almost any element can be extracted.
Raw data access
Full exports suit analysts who want to build their own analysis.
Industry standard
Widely documented, with many tutorials and community examples.
Local processing
Crawls run on your machine without a cloud project limit.
5. Limits to consider
Learning curve
The interface is data-dense and expects technical SEO knowledge.
Interpretation is manual
It reports conditions, and the auditor decides priority and meaning.
Hardware dependent
Large crawls need memory and disk space on your computer.
Reporting
Client-ready reports usually need extra work or other tools.
6. When to use Screaming Frog
Screaming Frog suits technical audits, migrations, custom data extraction, and analysts comfortable with spreadsheets. It is less suited to teams who want prioritized guidance, dashboards, or keyword and backlink research in one place.
7. Comparing with other tools
See the comparison pages for Screaming Frog vs Sitebulb vs SiteAuditLint, Screaming Frog vs Semrush vs SiteAuditLint, and Screaming Frog vs Ahrefs vs SiteAuditLint, as well as the guide to the best desktop SEO crawler for Windows.
8. Practical exercise: Run a configured crawl
Screaming Frog checklist
0 of 8 tasks completed
Key takeaways
- Screaming Frog SEO Spider is a desktop crawler that runs on your own computer.
- It offers deep configuration, rendering, custom extraction, and API integrations.
- List mode suits migrations and URL list checks.
- It provides raw data, and the auditor supplies interpretation and priority.
- It is strongest for technical analysts and flexible data work.
Knowledge check
Next, see a crawler that emphasizes prioritized hints and visualizations in the Sitebulb lesson.