Most sites do not have a crawl budget problem, and most articles about crawl budget will not tell you that.
Google’s own guidance is that sites with fewer than a few thousand URLs are generally crawled efficiently and do not need to think about this. If that is you, the honest answer is to go and fix something else.
This guide covers who actually has the problem, how to confirm it from your own data rather than from a blog post, and what to do if you do.
What crawl budget is
Two things combined.
Crawl capacity. How much Googlebot can fetch from your server without degrading it. If your server responds slowly or returns errors, Google backs off. If it responds fast and clean, Google is willing to fetch more.
Crawl demand. How much Google wants to crawl you. Driven by how popular your URLs are, how often they genuinely change, and how much of your site Google considers worth revisiting.
Crawl budget is roughly the overlap. You influence capacity through server performance, and demand through content quality and internal linking.
Do you actually have the problem?
Four tests. You need at least two.
|
Test |
What it means |
|
More than roughly 10,000 URLs |
Scale alone makes crawl allocation a real constraint |
|
URLs generated from templates with parameters or filters |
The URL count can be far higher than your page count |
|
New pages take weeks rather than days to appear in search |
Google is not reaching them promptly |
|
Search Console shows a large “Discovered, currently not indexed” group |
Google knows the URLs exist but is not spending crawl on them |
If none of these apply, stop here. You have a different problem, and our technical SEO guide covers the ones that are likelier.
If two or more apply, keep reading.
Confirm it from log files, not from a crawler
This is the part most audits skip, and it is the only part that proves anything.
A crawler like Screaming Frog tells you what a crawler can reach. Log files tell you what Googlebot actually fetched. Those are different, and the gap between them is where the problem lives.
Pull 30 days of server logs and filter to verified Googlebot requests. Verify by reverse DNS lookup rather than by user agent string, because the user agent is trivially spoofed and a meaningful share of “Googlebot” traffic in raw logs is not Google.
Then answer four questions:
What share of crawl goes to pages that can earn revenue? Group requests by URL pattern. If product and category pages are a minority of crawl while filters and parameters dominate, you have found it.
What is Googlebot’s average response time? Slow responses reduce crawl capacity directly.
What status codes is Googlebot getting? A high proportion of 404s, 301 chains or 5xx responses wastes budget on URLs that return nothing useful.
Which important pages were never crawled in 30 days? Compare your sitemap against the log. Anything commercially important and absent is the priority.
Where crawl budget actually goes
In order of how much we usually find wasted.
Faceted navigation and filters. One category with six filters that combine freely can generate thousands of URLs from one page of products. This is the single biggest source of waste on ecommerce sites.
URL parameters. Session IDs, tracking parameters, sort orders and pagination variants. Each unique string is a unique URL to Google.
Redirect chains. Every hop is a separate fetch. A three-hop chain costs three times what a direct redirect costs.
Soft 404s. Pages returning 200 with no real content. Empty search results and out-of-stock pages are the usual culprits. Google keeps crawling them because they look valid.
Thin generated pages. A page exists because a database row exists, not because anyone searches for it. Location pages for areas you do not serve, tag archives with one item, author pages for staff who wrote once.
Non-canonical duplicates. Covered in detail in our guide to duplicate content and page cannibalisation.
The fixes, in order of return
1. Stop generating the URLs
Always the best fix. If filter combinations do not need to be linkable, do not make them links. Use a form or JavaScript that does not produce crawlable URLs, rather than producing them and then trying to exclude them.
2. Block with robots.txt, carefully
robots.txt prevents crawling. This is the right tool for parameter patterns and filter URLs that should never be fetched.
Disallow: /*?sort=
Disallow: /*?sessionid=
Disallow: /search/
The critical thing to understand: robots.txt blocks crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, because Google knows it exists but cannot see its contents. If you need a URL out of the index, you must let Google crawl it and find a noindex, which means not blocking it in robots.txt.
Those two tools are mutually exclusive. Blocking a page you have also noindexed means Google never reads the noindex.
3. Fix status codes
Resolve redirect chains to single hops. Return proper 404s for pages that are gone rather than redirecting everything to the homepage, which is a soft 404 at scale. Fix 5xx errors, which directly reduce crawl capacity.
4. Use canonical tags correctly
For duplicates that must remain accessible, canonical tags consolidate signals. They do not save crawl budget, because Google still fetches the page to read the tag, but they prevent the duplication from fragmenting your rankings.
5. Improve internal linking
Pages buried five clicks from the homepage get crawled less. If a commercially important page has almost no internal links pointing at it, Google reads that as a signal it is unimportant. Linking rules belong in the template rather than in manual edits, which is how large sites keep this consistent.
6. Make the server faster
Crawl capacity responds directly to server speed. This overlaps with the user-facing work in website speed optimisation and Core Web Vitals, so it is rarely wasted effort.
What does not help
Worth listing, because these come up constantly.
Crawl-delay in robots.txt. Google ignores it.
Submitting the sitemap repeatedly. It does not increase crawl rate.
The URL Inspection tool at scale. Useful for one urgent page, not for a thousand.
Raising crawl rate in Search Console. The setting only lets you lower it. You cannot ask for more.
Nofollow on internal links. It does not conserve crawl budget the way it did years ago.
A realistic measurement cycle
Crawl changes are slow. Set expectations accordingly.
- Baseline from 30 days of logs, segmented by URL pattern.
- Ship one change set, not five, so you can attribute the result.
- Wait at least 30 days. Crawl patterns shift gradually.
- Re-pull logs, compare the same segments.
- Check indexed ratio per template in Search Console: pages indexed divided by pages submitted.
The metric to watch is not total crawl requests. It is what share of crawl reaches pages that can earn money. Total requests can fall while the outcome improves, because you removed waste.
Frequently asked questions
What is crawl budget?
Crawl budget is the number of URLs Googlebot will fetch from your site in a given period. It combines crawl capacity, which is how much your server can handle without degrading, and crawl demand, which is how much Google considers your URLs worth fetching.
Does my site have a crawl budget problem?
Probably not if you have fewer than a few thousand URLs. Google’s guidance is that smaller sites are generally crawled efficiently. The problem becomes real with large template-driven sites, heavy faceted navigation, new pages taking weeks to index, or a large “Discovered, currently not indexed” group in Search Console.
How do I check my crawl budget?
Server log files filtered to verified Googlebot requests over 30 days. Verify Googlebot by reverse DNS rather than user agent, since the user agent is easily spoofed. A site crawler shows what is reachable, not what Google actually fetched.
Does robots.txt save crawl budget?
Yes for crawling, but it does not remove pages from the index. A URL blocked in robots.txt can still appear in results if other pages link to it, because Google knows it exists but cannot read it. If you need a page out of the index, allow crawling and serve a noindex instead.
Do canonical tags save crawl budget?
No. Google must still fetch the page to read the canonical tag. Canonicals consolidate ranking signals between duplicates, which is valuable, but they do not reduce crawling.
Can I ask Google to crawl my site more?
No. Search Console lets you limit crawl rate but not increase it. You influence crawl indirectly, by improving server response times, removing waste, and giving Google reasons to want your pages.
Why are my new pages taking weeks to index?
Usually crawl allocation going elsewhere, commonly to filter and parameter URLs. It can also be weak internal linking, so new pages sit several clicks deep with few links pointing at them. Check the logs before assuming either.
Does page speed affect crawl budget?
Yes, directly. Faster server responses raise crawl capacity because Google can fetch more without degrading your site. Slow responses and 5xx errors cause Google to back off.
When this is worth paying attention to
For a 200-page business site, crawl budget is a distraction. For a marketplace, a multi-brand retailer or any site where templates generate tens of thousands of URLs, it is often the constraint that explains why good content is not ranking: Google has not reached it.
The work is specification work, not page-by-page editing. Our enterprise SEO service covers it with log analysis and template specs, and enterprise SEO strategy covers the operating model around it. If you are not sure which side of the line you are on, a website performance audit answers that first.
