robots.txt
robots.txt is the plain text file at a site’s root telling bots which addresses they may crawl. It manages crawling and alone does not keep a page out of the index: a blocked address that other sites link to can still appear in results, unread. To remove a page from the index, use noindex.
- Area
- SEO
- In Turkish
- robots.txt
- Read more · Guide
- Technical SEO checklist: 9 steps with free tools
Why it matters
A single wrong line can close the whole site to crawling. A block left over from a test environment after launch is one of the most common mistakes. The file is not a privacy tool either, since anyone can read it. It guides crawling and should not be used to hide sensitive information.
Example
Say a company launches its new site but forgets to remove the test-period line that blocks everything. Two weeks later organic visits fall close to zero. After the line is deleted, bots return and the pages are crawled again within days.
Related terms
- noindexnoindex is the directive that asks search engines not to show a page in results.
- llms.txtllms.txt is a proposed file, placed at a site’s root, that summarises the site’s most important pages in Markdown for language models.
- Canonical tagA canonical tag is the link element that tells search engines which address counts as the main page when the same or very similar content opens at more than one URL.
- Crawl budgetCrawl budget is the number of pages a search engine bot can and wants to crawl on a site in a given time.
- Google Search ConsoleGoogle Search Console is Google’s free tool that shows how a site is doing in Google Search.
- hreflanghreflang is the annotation that tells search engines where a page’s other language or country versions live, so a Turkish search gets the Turkish page and an English search the English one.
The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch
From our journal
From what we have written on SEO; they show how the term turns up in the work.
- 301 redirects and 404 errors: how to manage them301 for a page that moved, 404 or 410 for one that is gone: how to decide, how to sort 404s in Search Console, and what a useful 404 page contains.
- Canonical tags and duplicate content: how to fix itA step-by-step way to see which address Google picks when one page opens at several URLs, and to state your preference with the right signals.
- Core Web Vitals explained: how to read LCP, INP and CLSThe three thresholds, the gap between field and lab data, and which row to look at in Search Console and PageSpeed Insights, step by step.
- How to brief a digital agency: seven headings on one pageThe one-page document to write before you ask for proposals: goal, current numbers, audience, what has been tried, constraints, who decides and the deadline. Plus a template and how to compare proposals.