Crawl budget is the number of URLs a search engine crawler such as Googlebot will fetch and process on your site in a given window. Google describes it as the product of crawl capacity (how much your server can handle without slowing down) and crawl demand (how often Google thinks your URLs are worth refetching).
Why it matters
For small sites under a few thousand URLs, crawl budget is rarely a concern — Google can keep up easily. For large e-commerce, news, or programmatically generated sites, it becomes critical: if Googlebot spends its budget on faceted-navigation noise, internal search results, or duplicate parameters, your important pages get crawled less often and new content takes longer to index. Slow servers compound the problem because Google throttles crawl rate when responses get slow. Wasted budget directly delays how quickly product changes, prices, and new articles appear in search.
How to check
- Pull server logs (or Search Console's Crawl Stats report) and segment Googlebot requests by URL pattern to see where budget is going.
- Block low-value URL patterns — pagination beyond a depth, sort orders, internal search — via robots-txt.
- Apply noindex to thin pages you must keep accessible, but never block them in robots.txt at the same time.
- Consolidate duplicates with a canonical-tag so signals merge instead of fragmenting.
- Eliminate redirect chains; each redirect hop costs an extra fetch.
- Submit a clean sitemap-xml of canonical, indexable URLs only.
- Improve TTFB and server response time so each fetch costs Googlebot less.
