Crawl Budget: When It Matters and How to Manage It

If your site has a few hundred pages, you can probably skip this article. Google has said plainly that crawl budget is mainly a concern for very large sites or sites with many pages changing frequently. For everyone else, Google will crawl what's worth crawling without help.

But if you run a marketplace with 200,000 listings, an ecommerce store with layered filters, or a job portal where pages appear and expire daily, crawl budget can decide whether new pages show up in search within days or months.

What crawl budget actually is

Google describes it as a combination of two things:

• Crawl capacity limit. How many requests Googlebot can make to your site without overloading the server. If your server responds quickly and without errors, the limit rises. If it slows down or returns 5xx errors, Googlebot backs off.
• Crawl demand. How much Google wants to crawl your URLs, based on how popular they are and how often they change. Pages nobody links to and that never change get visited rarely.

Your effective crawl budget is the number of URLs Google can and wants to crawl in a given period. You can't buy more of it, but you can stop wasting it.

Signs it's a problem for you

• Search Console's Pages report shows a large and growing number of URLs under "Discovered, currently not indexed", meaning Google knows about them but hasn't fetched them.
• New product or listing pages take weeks to get crawled, even though they're in your sitemap.
• The Crawl Stats report (Settings > Crawl stats in Search Console) shows Googlebot spending most of its requests on parameter URLs, redirects or 404s rather than real pages.
• Your site generates far more URLs than it has unique pages. A store with 5,000 products and 400,000 crawlable URLs has a problem.

If none of these apply, your time is better spent on content or links.

Where crawl budget gets wasted

Faceted navigation. A clothing store lets shoppers filter by size, colour, fabric, price and brand. Every combination creates a URL. Five filters with a handful of values each can produce hundreds of thousands of near-duplicate pages, and Googlebot will happily try to crawl them all.

Sorting and tracking parameters. ?sort=price-asc, ?utm_source=..., ?sessionid=.... Each is a new URL to a crawler.

Infinite spaces. Calendars with "next month" links that go on forever, or internal search pages that generate results for any query.

Redirect chains and soft 404s. Every hop is another request. Pages that say "no products found" but return 200 look like real pages and get recrawled.

Slow servers. If each page takes two seconds to generate, Googlebot fetches fewer of them.

How to fix it

Control faceted URLs

Decide which filter combinations deserve to be indexable pages. "Men's running shoes" probably does. "Men's running shoes, size 9, blue, under 3000, sorted by newest" does not. Then:

• Block crawling of the low-value parameter patterns in robots.txt (see the robots.txt guide).
• Avoid normal crawlable links to unwanted combinations; some stores apply filters with JavaScript or form submissions instead.
• Use canonical tags for variants that must stay crawlable, knowing canonicals don't stop crawling by themselves.

Robots.txt is the tool that actually saves crawl budget, since noindex and canonical pages still have to be crawled for Google to read them.

Clean up status codes

Fix internal links that point to redirects so they point at the final URL. Return a real 404 or 410 for removed pages rather than a soft 404. Keep 5xx errors near zero. Fixing 404 errors covers how to decide which dead URLs to redirect.

Keep the sitemap accurate

List only indexable, canonical URLs, and use accurate lastmod dates so Google can prioritise changed pages. Splitting sitemaps by section lets you see which areas are lagging.

Improve server speed

Faster responses raise the crawl capacity limit. Caching, a CDN for static assets and enough server resources all help. Watch the average response time in Crawl Stats.

Strengthen internal linking to important pages

Crawl demand follows importance. New listings linked from the homepage or a "latest" section get found faster than pages buried six clicks deep. External links to key sections also raise demand.

Remove or consolidate thin pages

Thousands of near-empty tag pages, expired listings with no content, or auto-generated location pages dilute the site. Removing or noindexing them (and eventually blocking them) lets Google focus on what matters.

A note on log files

Crawl Stats gives a summary, but server logs show every Googlebot request: which URLs, how often and with what status. On a large site, that's the clearest view of where crawl activity really goes. Log file analysis explains how to get started.

What doesn't help

Submitting URLs one by one through the URL Inspection tool won't scale and has daily limits. Changing the crawl rate setting isn't possible any more; Google retired the crawl rate limiter in Search Console. And adding more pages to "show Google you're active" usually makes the problem worse.

Open the Crawl Stats report and look at the breakdown by response and by file type for the last 90 days. If a large share of requests go to redirects, 404s or parameter URLs, you've found where to start.