Most sites worrying about crawl budget do not have a crawl budget problem. Google has said this plainly and repeatedly: if your site has under a few thousand URLs, it will generally be crawled fine, and your time is better spent almost anywhere else.
So the first useful thing this article can do is help you find out whether you are in that group and stop reading.
Do You Actually Have a Problem?
Three conditions. You need at least one.
- Scale. Tens of thousands of URLs or more, especially if generated: faceted navigation, filters, search results, large catalogues.
- Velocity. Content that changes frequently and needs recrawling fast to be worth anything. News, listings, inventory, pricing.
- Observed symptoms. Pages you care about taking weeks to get indexed, or updates taking weeks to be reflected.
None of those? You do not have a crawl budget problem. You may well have an indexing problem, which looks identical from the outside and has an entirely different cause. Pages that are crawled and then not indexed are usually being judged as not worth indexing: thin, duplicative, or low value. No amount of crawl optimisation fixes that, and treating it as a crawl issue wastes months.
That distinction is the single most common misdiagnosis I see. Check whether the page was crawled at all before assuming anything.
What Crawl Budget Is Made Of
Two components, and they behave differently.
Crawl capacity is how much Google is willing to request without degrading your server. If your responses slow down or start erroring, this drops. If they are fast and clean, it rises. This one you influence through hosting and performance.
Crawl demand is how much Google actually wants your pages, driven by popularity, freshness, and whether previous crawls found anything worth having. This one you influence by being worth crawling.
Sites obsess over the first and neglect the second. A fast server crawling 50,000 worthless URLs does not help you. Demand is the ceiling; capacity only determines how fast you reach it.
Where It Actually Leaks
In practice, wasted crawl is nearly always one of these.
Leak | What to do |
|---|---|
Faceted navigation generating near-infinite combinations | Block the parameter combinations that add nothing in robots.txt. Do not rely on canonicals alone, the URL still gets crawled before the canonical is read. |
Internal search result pages | Block in robots.txt. These have no business being crawled at any scale. |
Redirect chains | Point every hop directly at the final destination. Each hop is a request spent learning nothing. |
Session IDs and tracking parameters in URLs | Strip them, or block the parameter. |
Deep pagination on low-value archives | Reduce depth, or stop linking past a sensible point. |
Soft 404s returning a 200 | Return an actual 404 or 410. A 200 on an empty page invites repeat crawling forever. |
Worth being precise about one thing, because it is widely got wrong: noindex does not save crawl budget. The page still has to be fetched for the directive to be read. robots.txt prevents the fetch. If your goal is saving crawl, only the second one does that, and the tradeoff is that a blocked URL cannot pass signals or be deindexed through a directive it is not allowed to read.
Read Your Logs, Not Your Assumptions
Everything above is guesswork until you look at server logs. They are the only source that tells you what was actually requested, by whom, and what it received.
Four questions worth answering:
- What proportion of crawl requests hit URLs you actually want indexed? Under half means something is consuming the rest and you can go find it.
- What status codes are crawlers receiving? A meaningful share of 3xx or 4xx is budget spent on nothing.
- Which sections get crawled most? Compare against which sections make money. They are often not the same.
- How long since your important pages were last crawled? That number is your real refresh latency.
A single week of logs usually answers all four and frequently produces one obvious, embarrassing finding: an entire parameter space being crawled, or a redirect loop nobody knew existed.
The New Traffic Nobody Budgeted For
Something has changed here, and it changes the arithmetic.
AI crawlers now make up a substantial and growing share of automated traffic on many sites. They are far less well behaved than Googlebot: they honour crawl-delay inconsistently, re-request aggressively, and do not adjust politely when your server slows.
This creates a genuine conflict. That traffic consumes real server capacity, and reduced capacity means Googlebot crawls you less. Blocking AI crawlers protects performance but removes you from the systems increasingly used to answer questions. Allowing them costs resources and may return nothing.
There is no universally correct answer, and anyone telling you there is has not looked at your logs. The defensible position for most sites is selective: allow the crawlers belonging to systems that actually cite sources and send referral traffic, rate-limit or block the ones that take content and give nothing back, and make that decision from your own data rather than a blog post. If you want the visibility side of that argument, my write-up on generative engine optimization covers what you gain by being readable.
The Order I Would Work In
- Confirm you have a crawl problem rather than an indexing problem
- Pull a week of logs and find where requests are going
- Block the obvious waste: internal search, useless parameters, infinite facets
- Flatten redirect chains
- Fix soft 404s
- Improve response times, which raises capacity
- Decide your AI crawler policy deliberately
- Re-pull logs and measure whether the ratio improved
Steps one and two are most of the value. If they reveal that your crawl is already going almost entirely to pages you care about, you are done, and that is a good outcome rather than a wasted afternoon.
Pages taking weeks to get indexed?
I will read your logs and tell you whether it is genuinely crawl budget or something being judged not worth indexing, because the fixes are completely different.
📞 Book a free 20-minute review, or see full website SEO.
Serving all 50 US states, remote. ✉ info@shazzseo.com
Written by Shahzaib Ul Hassan, senior AI SEO consultant and founder of ShazzSEO. Ranking sites since 2009. 500+ websites optimized, 3,000+ students trained.