robots.txt

robots.txt is the plain text file at a site’s root telling bots which addresses they may crawl. It manages crawling and alone does not keep a page out of the index: a blocked address that other sites link to can still appear in results, unread. To remove a page from the index, use noindex.

Area
SEO
In Turkish
robots.txt

Why it matters

A single wrong line can close the whole site to crawling. A block left over from a test environment after launch is one of the most common mistakes. The file is not a privacy tool either, since anyone can read it. It guides crawling and should not be used to hide sensitive information.

Example

Say a company launches its new site but forgets to remove the test-period line that blocks everything. Two weeks later organic visits fall close to zero. After the line is deleted, bots return and the pages are crawled again within days.

The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch

From our journal

From what we have written on SEO; they show how the term turns up in the work.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.