Crawl Rate: How Googlebot Sets It and How to Improve It

Crawl rate is the number of requests a search engine crawler, such as Googlebot, makes to your website over a given period, including how many connections it opens at once and how long it waits between fetches. Google sets it automatically based on your server's health and how much it wants to crawl your content.

A website that responds quickly and consistently can support more crawling. When a server slows down, returns errors, or struggles with incoming requests, Googlebot reduces its activity to avoid overloading the host.

Understanding crawl rate helps you diagnose crawling problems, protect server resources, and make important pages easier for search engines to discover. This guide covers how Google sets crawl rate, how to measure it, why it drops, and what you can actually do about it.

What Is Crawl Rate?

Crawl rate describes the speed at which a search engine crawler requests URLs from a host. For Google, that means the number of simultaneous connections Googlebot uses and the time it waits between requests.

Google does not assign a permanent, fixed crawl rate to any website. Its crawling infrastructure adjusts continuously based on server performance and the amount of content it wants to crawl.

Consider two websites:

• Website A responds quickly and consistently, with very few server errors.
• Website B frequently experiences slow responses, timeouts, and 5xx errors.

Googlebot can crawl Website A more efficiently, while it slows down on Website B to avoid adding pressure to an already struggling server.

How Does Googlebot Determine Crawl Rate?

Google balances two things: how much crawling your server can handle (crawl capacity) and how much Google wants to crawl your site (crawl demand).

Crawl rate limit (crawl capacity limit)

Google originally called this the crawl rate limit. Its current documentation calls it the crawl capacity limit. Both terms describe the maximum number of parallel connections and requests Googlebot will use on a host without degrading the server for real users.

Google considers signals such as:

• Server response time and Time to First Byte (TTFB)
• Host availability, including successful robots.txt and DNS fetches
• Connection stability
• HTTP response codes, especially 5xx and 429
• How many simultaneous connections the server handles without slowing down

When a server responds quickly and reliably, the limit can rise over time. When response times climb or errors become frequent, Googlebot lowers the limit. Repeated HTTP 5xx or HTTP 429 (Too Many Requests) responses are clear signals that the server is struggling.

The limit is applied per host. If shop.example.com returns errors, Googlebot can slow down across that entire hostname, not just the failing URLs.

Crawl demand

Crawl demand is how much Google wants to crawl your site. A server that could handle thousands of requests per minute will not receive them unless Google has a reason to crawl that much.

Crawl demand depends on:

• Perceived inventory: how many URLs Google knows about on the site
• Popularity: URLs that are more popular across the web tend to be crawled more often
• Staleness: how often content changes and how outdated Google's stored copy is likely to be
• Duplicate and low-value URLs: large numbers of near-duplicate pages can waste demand
• Site-wide events: migrations, domain moves, and large URL changes temporarily raise demand

A news site that publishes dozens of articles a day has very different crawl demand from a small brochure site that changes twice a year, even if both run on identical servers.

Crawl Rate vs. Crawl Frequency vs. Crawl Budget vs. Crawl Ratio

These terms are related, but each describes a different part of crawling.

Many SEO sources use "crawl rate" loosely to mean both speed and frequency. Google's own documentation keeps them separate, and the distinction matters when diagnosing problems. A site can be crawled frequently while only a small share of its URLs is ever requested.

Crawl budgets mostly matter for large sites, roughly those with tens of thousands of frequently changing URLs or hundreds of thousands of pages. A large store with thousands of parameter-based filter URLs can spend much of its budget on pages that add no unique value.

Crawl ratio is a useful companion metric. To calculate it, compare the URLs found in a full site audit against the unique URLs Googlebot requested in your server logs over the same period. A low ratio on important sections usually points to internal linking or discovery problems.

How Is Crawl Rate Measured?

There is no single "crawl rate" number in any Google tool. Different sources describe it in different units:

• Requests per second: the instantaneous speed of fetching
• Parallel connections: how many requests Googlebot runs at the same time
• Requests per day: the total volume shown in Search Console

Crawl rate is technically the first two. The Crawl Stats report in Google Search Console shows the third, as daily totals. To see per-second behavior, analyze timestamps in your server access logs.

What Factors Affect Googlebot Crawl Rate?

1. Server response time

Slow responses directly reduce how many pages Googlebot can fetch in the same period. Google's crawl budget documentation notes that consistently slow average response times can lower the crawl capacity limit for a host. Watch the average response time in Crawl Stats and investigate any sudden rise.

2. HTTP status codes

Status codes tell Googlebot what happened with each request:

Repeated 5xx and 429 responses cause Google to slow down. Not every 4xx is a problem: a 404 or 410 for a permanently removed page is the correct response.

3. URL structure and crawl traps

Some site features generate far more URLs than you have real pages:

• Faceted navigation and filter combinations
• Sorting and view parameters
• Session IDs in URLs
• Calendars that generate endless date pages
• Tracking parameters and duplicate URL variations

These crawl traps inflate the URLs Google discovers and requests. Control how they are generated so crawlers cannot follow an endless chain of variations.

4. Internal linking

Internal links are how crawlers find pages. Important pages with few internal links, or orphan pages with none, are harder to discover through normal crawling. Link to priority pages from relevant, crawlable HTML links.

5. Redirect chains

Each hop in a chain costs an extra request. Googlebot follows up to 10 hops before giving up on that fetch.

# A two-hop chain
/old-page/  >  /page-v2/  >  /page-final/

# Fix: link and redirect straight to the final URL
/old-page/  301  /page-final/

Update internal links to point straight at the final URL. See the redirect chain issue and our guide to finding and fixing redirect problems.

6. Duplicate content and URL variations

Multiple URLs serving the same content multiply crawling work. Common culprits are HTTP and HTTPS versions, www and non-www hosts, trailing slash variations, and duplicate category or filter pages. Pick one version, redirect the others, and use consistent canonical tags.

7. Firewalls, CDNs, and bot protection

Security rules can block or throttle Googlebot without anyone noticing. A WAF challenge, a CDN bot-management rule, or an aggressive rate limiter can return 403, 429, or 503 responses to Google while real users see the site normally. Related reading: how to diagnose search engine crawling problems on Cloudflare.

How to Check Googlebot Crawl Rate in Search Console

The Crawl Stats report in Google Search Console shows how Google has crawled your site over the last 90 days.

1. Open Google Search Console.
2. Select a domain-level or root-level property.
3. Go to Settings.
4. Under Crawling, open the Crawl stats report.

The report shows total crawl requests, total download size, and average response time over time. It also includes host status and four breakdowns of crawl requests:

• By response: 200, 301, 404, 5xx, and other status codes
• By file type: HTML, JavaScript, CSS, images, and more
• By purpose: discovery (URLs Google had not crawled before) vs. refresh (known URLs being revisited)
• By Googlebot type: smartphone, desktop, image, video, AdsBot, and others

What to look for

Why Did My Crawl Rate Drop?

A crawl drop is not automatically an SEO problem. A small business site with a handful of pages that rarely change will naturally be crawled less often, because Googlebot learns it does not need to check as frequently. A sudden drop matters more on sites that publish or update content constantly, such as news sites, blogs, and large stores.

Google's guidance groups the causes into three areas: technical problems on your site, blocked access, and lower crawl demand. Work through them in this order:

1. Host availability errors. Check Crawl Stats host status for robots.txt fetch failures, DNS errors, and connection failures.
2. Rising response times. Compare the drop with the average response time graph. Slow responses lower crawl capacity.
3. 5xx and 429 spikes. Look for server errors or rate limiting that began around the same date.
4. Firewall, CDN, or bot protection changes. Confirm that a new WAF rule or bot-management setting is not challenging or blocking Googlebot.
5. Recent migrations or infrastructure changes. New hosting, a CDN switch, DNS changes, or HTTPS changes can temporarily disrupt crawling.
6. robots.txt changes. Make sure a recent edit did not accidentally disallow important sections.
7. Lower crawl demand. If nothing above applies, Google may have less reason to crawl. A drop after removing crawl traps is often a good sign.

If you find a technical cause and fix it, crawling usually recovers on its own as Google sees the server responding normally again.

When the cause is crawl demand

If host status is healthy, response times are stable, and nothing is blocked, the drop is likely a demand signal. Google has limited crawling resources and prioritizes sites that are popular, high quality, and updated often. If Google sees your content as less in demand or less useful than alternatives, it spends its crawling elsewhere.

There is no technical switch for this. The long-term fix is publishing helpful content that earns links and visits, and meaningfully updating important pages rather than changing dates without changing content.

If you still cannot find the cause, the Google Search Central Help Community is a good place to ask, with product experts and forums in many languages.

Can You Increase Google Crawl Rate?

Not directly. Google's Crawl Stats documentation states that you cannot tell Google to increase your crawl rate. What you can do is remove the technical limits holding it back and give Google more reasons to crawl. These steps improve crawl efficiency, which is what actually matters.

1. Improve server performance

• Use effective server-side and CDN caching
• Optimize slow database queries
• Make sure hosting resources match your traffic and crawler load
• Fix recurring server errors
• Track response time trends, not just one-off checks

2. Keep your XML sitemap clean

List only canonical, indexable URLs that return 200. Remove redirected, noindexed, and deleted pages. Use accurate lastmod values that reflect real content changes, because Google ignores lastmod when it is consistently unreliable.

3. Control unnecessary URL generation

Audit filters, sorting, tracking parameters, and other features that create URLs. Use robots.txt to block crawl traps that offer no unique content, but do this carefully, since blocked URLs cannot pass signals through crawling.

4. Fix broken internal links

Update links that point to 404s, wrong destinations, or redirects so every internal link goes directly to a live, final URL.

5. Remove redirect chains

Replace multi-hop chains with a single redirect and point internal links at the destination.

6. Review canonical tags

Make sure canonical tags point to the preferred version of duplicate or near-duplicate pages. Canonicals are hints, not crawl directives, so pair them with consistent internal linking.

7. Strengthen internal links to priority pages

Pages that matter most should be close to the homepage and linked from relevant sections. This raises their perceived importance and makes them easier to discover.

8. Publish and update useful content

Crawl demand rises when a site regularly adds or meaningfully updates content people link to and visit. Technical fixes remove limits, but content gives Google a reason to come back.

Should You Reduce Googlebot Crawl Rate?

In most cases, no. Google adjusts automatically based on your server's health and its crawl demand. Only consider reducing crawl rate when Googlebot is causing a real infrastructure problem, and confirm through your logs that the traffic is actually from Google first.

What happened to the Search Console crawl rate limiter tool?

Search Console used to have a crawl rate limiter setting that let site owners ask Google to crawl more slowly. Google retired it on January 8, 2024, saying its automated systems now adjust crawl rate well enough on their own. Older guides that tell you to use this setting are out of date.

The current emergency option

Google's Reduce Google Crawl Rate documentation describes one short-term method: return HTTP 500, 503, or 429 responses to Googlebot requests. Googlebot slows down when it sees many of these errors.

This has real costs:

• Crawling can slow down across the entire hostname, not just the affected URLs
• New pages take longer to be discovered
• Updated content, prices, and stock status take longer to refresh in Search
• Removed pages may stay indexed longer
• URLs that keep returning these errors for several days can be dropped from the index

Google recommends this only for brief emergencies, from a few hours up to one or two days. If heavy crawling continues and serving errors is not practical, Google provides a form to report unusually high crawling.

Crawl Rate Limits for Bingbot, CCBot, and AI Crawlers

Googlebot is often not the main source of crawler load. Before changing anything, check your access logs to see which user agents are sending the most requests.

CCBot is the crawler used by Common Crawl, a nonprofit that builds a public web archive used widely in research and AI training. You can slow it with crawl-delay or block it in robots.txt. Rate limiting or blocking it has no effect on Google Search.

For AI crawlers that ignore crawl-delay, use server-level or CDN rate limiting by user agent. Make sure those rules never match Googlebot. If AI crawlers are being blocked when you do want them, see how to fix 403 errors blocking AI crawlers.

How to verify that a crawler is really Googlebot

Many scrapers fake the Googlebot user agent, so a log line that says "Googlebot" proves nothing on its own.

1. Run a reverse DNS lookup on the requesting IP address.
2. Confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com.
3. Run a forward DNS lookup on that hostname and confirm it resolves to the original IP.

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

You can also match IPs against the JSON lists of crawler IP ranges that Google publishes. Fake Googlebot traffic can be blocked safely, which often fixes "Googlebot overload" problems without touching real crawling.

Common Crawl Rate Mistakes to Avoid

How SiteAuditLint Helps Identify Crawling Problems

Crawl Stats tells you how Google is crawling. It does not tell you which site issues are making that crawling less efficient. SiteAuditLint is a desktop SEO auditing tool that finds those issues so you can fix them.

• Broken links: internal links pointing to URLs that fail
• Redirect chains: multi-hop redirects you can collapse into one
• HTTP status codes: 4xx pages, 5xx errors, and unexpected responses
• Canonical tags: missing, conflicting, or incorrect canonicals
• XML sitemaps: non-canonical, redirected, or broken sitemap URLs
• Internal linking: orphan pages and important pages with weak link support
• Robots directives: robots.txt rules and meta robots tags that block or restrict pages
• Historical comparisons: issues introduced after a release or migration

SiteAuditLint's full URL inventory also gives you the baseline for crawl ratio: compare it with the URLs Googlebot requested in your server logs to see which sections are being missed. Audit comparison shows which crawl issues are new after a release, and scheduled audits catch regressions before they show up as a crawl drop.

SEO crawlers have their own crawl rate too. When auditing a live production site, use a conservative request speed and thread count so the audit itself does not slow the server down for users or for Googlebot.

SiteAuditLint does not measure Google's actual crawl rate and does not replace Search Console or log analysis. It works alongside them by showing the technical causes behind what those tools report.

Crawl Rate FAQ

What does crawl mean in SEO?

Crawling is the process of search engine bots requesting pages from a website to discover and read their content. A page generally has to be crawled before it can be indexed and shown in search results.

What does crawl budget mean?

Crawl budget is the set of URLs Google can and wants to crawl on a site. It combines the crawl capacity limit (what the server can handle) with crawl demand (how much Google wants to crawl).

How do I improve crawl rate?

You cannot force Google to crawl faster. Improve server response times, fix 5xx errors, remove crawl traps and redirect chains, keep your XML sitemap clean, and strengthen internal links to important pages.

What is a good crawl rate?

There is no universal target. A healthy crawl rate is one where Googlebot keeps up with your new and updated pages without causing server slowdowns. Judge it by how quickly important pages are discovered and refreshed, not by request totals.

Does crawl rate affect rankings?

No. Crawl rate is not a ranking factor. It affects how quickly Google can discover and refresh content, which can matter for time-sensitive pages.

Does Googlebot respect crawl-delay?

No. Googlebot ignores the crawl-delay directive in robots.txt. Bingbot and CCBot do support it.

Final Thoughts

The goal is not to maximize Googlebot requests. It is to make sure Google spends its crawling on the pages that matter, without straining your server.

Keep the server fast and stable, remove unnecessary URL variations, fix technical errors, and make important pages easy to reach through internal links. Monitor Crawl Stats and your server logs to see what Googlebot actually does, then use technical audits to find and fix what gets in its way.