The two parts
We read Google's page on managing crawl budget on 9 October 2026. It says the budget depends on two things.
- Crawl capacity limit: how much of your server's time Google will use. Every site starts with a cautious limit. The limit goes up when your site answers fast and steadily. It goes down when the site slows, returns server errors, or tells Google to slow down.
- Crawl demand: how much Google wants to crawl. For Googlebot, this depends on site size, how often pages change, page quality and relevance. The part you control most is the list of addresses Google knows about. Duplicates and unwanted addresses waste crawl time.
Google treats each hostname as a separate site, so a subdomain has its own budget. A hostname is the name of the site, such as shop.example.com.
Does your site need to care?
Google says its guide is for large or fast-changing sites. It names three groups: sites with 1 million or more pages that change about once a week, sites with 10,000 or more pages that change every day, and sites with many addresses marked Discovered, currently not indexed in Search Console. Google says these are rough estimates, not exact thresholds.
Google says smaller sites generally do not need the guide, particularly if new pages get crawled the day they are published. For these, Google says keeping your sitemap current and checking the Page indexing report in Search Console from time to time is adequate.
What Google says helps
- Merge duplicate content so Google spends time on unique pages. See canonical tag.
- Use robots.txt to block addresses that do not matter. Google says not to use noindex for this, because Google still requests those pages. It adds that blocking frees no budget for other pages unless you are already at your capacity limit.
- Return a 404 or 410 error for pages that are gone for good. Google calls a 404 a strong signal not to crawl that address again.
- Fix soft 404s. These are pages that look like errors but do not say so, and they keep being crawled.
- Keep sitemaps current and add a last-modified date for updated pages.
- Avoid long redirect chains. See 301 redirect.
- Make your pages answer quickly.
A made-up example
A mobile accessories shop in Surat has 800 product pages. Its filters for colour, price and brand also create thousands of near-copy addresses.
By Google's rough sizes, the 800 real pages make a small site. The shop's job is not to chase a bigger budget. It is to keep the sitemap current, check the Page indexing report, and cut down the filter addresses that repeat the same content. If the report lists many addresses as Discovered, currently not indexed, the shop is in the third group Google names, and the guide is worth reading closely.
Common mistakes
- Worrying about crawl budget on a small site that Google crawls without trouble.
- Blocking pages with noindex and expecting it to save crawl time.
- Leaving old test pages, filter addresses and duplicates open.
- Mixing it up with ranking. Google's page is about which pages get fetched. It does not talk about where they rank.
Related terms
- Indexing: when a search engine stores a page so it can show it in results.
- Sitemap: a file that lists the pages you want found.
- Robots.txt: a file that tells crawlers which parts of a site not to fetch.
- Technical SEO (search engine optimisation): the setup work that lets search engines read your site.
- Internal linking: links between pages of the same site.
To see how Google visits a site, read our Googlebot crawling guide. For a full check of your setup, see our technical SEO work.
Sources
- Google Search Central, "Managing crawl budget for large sites" (developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget), last updated 22 July 2026, read 9 October 2026.
