Crawl budget
Crawl budget is the number of pages a search engine bot can and wants to crawl on a site in a given time. Server capacity and demand for the pages set it together. It is rarely a problem for small and mid-sized sites and matters on very large, filtered or frequently updated ones.
- Area
- SEO
- In Turkish
- Tarama bütçesi
- Read more · Guide
- Technical SEO checklist: 9 steps with free tools
Why it matters
A bot crawls only a limited number of pages on a site. If that share goes to filter pages, duplicates or broken addresses, important pages are discovered and refreshed late. New products or posts enter the index slowly. On large sites, cutting needless addresses makes a real difference.
Example
Say a site has 5,000 real product pages and another 45,000 addresses created by filter combinations. Most of what the bot crawls each day is filter addresses. Once those are closed to crawling, the bot can spend its time on the 5,000 products and new items appear sooner.
Related terms
- Canonical tagA canonical tag is the link element that tells search engines which address counts as the main page when the same or very similar content opens at more than one URL.
- Google Search ConsoleGoogle Search Console is Google’s free tool that shows how a site is doing in Google Search.
- hreflanghreflang is the annotation that tells search engines where a page’s other language or country versions live, so a Turkish search gets the Turkish page and an English search the English one.
- IndexingIndexing is the step in which a search engine processes a crawled page and stores it in the database its results come from.
- Internal linkAn internal link is a link from one page of a site to another page of the same site.
- Meta descriptionA meta description is the HTML tag that sums up a page in a sentence or two and may be shown under the headline in search results.
The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch
From our journal
From what we have written on SEO; they show how the term turns up in the work.
- 301 redirects and 404 errors: how to manage them301 for a page that moved, 404 or 410 for one that is gone: how to decide, how to sort 404s in Search Console, and what a useful 404 page contains.
- Canonical tags and duplicate content: how to fix itA step-by-step way to see which address Google picks when one page opens at several URLs, and to state your preference with the right signals.
- Core Web Vitals explained: how to read LCP, INP and CLSThe three thresholds, the gap between field and lab data, and which row to look at in Search Console and PageSpeed Insights, step by step.
- How to brief a digital agency: seven headings on one pageThe one-page document to write before you ask for proposals: goal, current numbers, audience, what has been tried, constraints, who decides and the deadline. Plus a template and how to compare proposals.