Crawl budget

Crawl budget is the number of URLs a search engine’s crawler is willing and able to fetch from a site in a period. Google describes it as the combination of crawl capacity (how fast the server can be crawled without harm) and crawl demand (how much the crawler wants the site’s pages, driven by popularity and staleness).

Who needs to care

Google’s guidance is explicit that crawl budget is a concern for large sites, roughly a million or more unique pages, or medium sites with content that changes daily. A few hundred pages that are all linked and all worth indexing will be crawled regardless. For such sites, the useful part of the idea is the list of things that waste crawls: faceted and parameter URLs, soft 404s, redirect chains, duplicate pages, and infinite spaces such as calendars.

AI crawlers

AI crawlers are not documented as having a budget concept, but the waste is the same: a crawler that spends its fetches on duplicates sees fewer real pages. Server logs show what each agent actually fetched. Our AI bot log analyser summarises a log by AI agent and path.