Crawl budget

Crawl budget is the number of pages a search engine bot can and wants to crawl on a site in a given time. Server capacity and demand for the pages set it together. It is rarely a problem for small and mid-sized sites and matters on very large, filtered or frequently updated ones.

Area
SEO

Why it matters

A bot crawls only a limited number of pages on a site. If that share goes to filter pages, duplicates or broken addresses, important pages are discovered and refreshed late. New products or posts enter the index slowly. On large sites, cutting needless addresses makes a real difference.

Example

Say a site has 5,000 real product pages and another 45,000 addresses created by filter combinations. Most of what the bot crawls each day is filter addresses. Once those are closed to crawling, the bot can spend its time on the 5,000 products and new items appear sooner.

The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch

From our journal

From what we have written on SEO; they show how the term turns up in the work.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.