How to prepare robots.txt and a sitemap: 6 steps

What the two files do and do not do, the rules Google supports, harmful lines, size limits, and submitting and testing them in Search Console.

· 13 min readSEO

Short answer

robots.txt tells Google which addresses it may crawl on your site; it does not remove a page from the index, and for that you need noindex. A sitemap lists your important addresses and does not guarantee indexing. This guide covers writing, publishing, submitting and testing both.

Picture this: on the day the new site goes live, the developer copies the robots.txt file from the test environment to the live server. The file holds two lines: User-agent: * and Disallow: /. The site opens and everything works, yet nothing is left that lets Google crawl it. This article is the practical answer to how to prepare robots.txt and a sitemap: you will write the two files in turn, publish them, submit the sitemap in Search Console and test the result. The rest of the site’s technical checks are in the technical SEO checklist; here we stay with these two files.

Two files, two different jobs

robots.txt (definition in the glossary) tells Google which addresses it may crawl on your site. According to Google’s documentation, its crawlers download and parse this file before they crawl a site. A sitemap lists the addresses you want crawled and shown in search. One is a crawling rule, the other is a list; neither removes a page from search, and neither guarantees indexing.

robots.txt and the sitemap do two different jobs robots.txt Manages crawling At the site root: example.com/robots.txt Does not remove a page from the index Sitemap Tells Google which URLs matter Recommended place: the site root Does not guarantee indexing To keep a page out of search: the noindex rule robots.txt and the sitemap do two different jobs robots.txt Manages crawling At the site root: example.com/robots.txt Does not remove a page from the index Sitemap Tells Google which URLs matter Recommended place: the site root Does not guarantee indexing To keep a page out of search: the noindex rule
Figure: robots.txt manages crawling and the sitemap tells Google which URLs matter; neither removes a page from the index, which takes the noindex rule.

If you want to keep a page out of search, Google’s documentation points to the noindex rule. It goes in the page’s <meta> tag or in an HTTP response header; once Google crawls the page and reads the rule, it drops the page from search results entirely. One detail matters: if robots.txt blocks the page, the crawler never sees the noindex rule, and the page can keep appearing in results, for example because other pages link to it. The same document also says that Google does not support a noindex rule written inside robots.txt.

Step 1: write the robots.txt rules

The file is plain text: write it in a text editor and save it as UTF-8. In Google’s robots.txt specification, four fields are supported: user-agent, allow, disallow and sitemap. The document says other fields, such as crawl-delay, are not supported. Here is an example file (the domain is a placeholder):

User-agent: *
Disallow: /admin/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Knowing how the rules are read prevents surprises in the file:

  • Groups. Each group starts with a User-agent line. According to Google, only one group is valid for a given crawler: the one that matches its name most closely. A group that names a crawler is not combined with the * group. So if you write a separate group for Googlebot, the rules in the * group are ignored for Googlebot and you must repeat what it needs in that group.
  • Default. Every address that is not blocked can be crawled.
  • Case. Field names (Disallow) are case-insensitive, while path values are case-sensitive: /Admin/ and /admin/ are different paths.
  • Wildcards. Google supports * (zero or more characters) and $ (the end of the URL).
  • Conflicts. Google uses the most specific rule, judged by the length of the path; where rules conflict, it uses the least restrictive one.

The example below shows path length and the $ rule together:

User-agent: *
Disallow: /private/
Allow: /private/press/
Disallow: /*.pdf$

Addresses under /private/press/ can be crawled, because the longer, more specific Allow rule wins. Addresses that end in .pdf are blocked; because $ marks the end of the URL, an address such as /file.pdf?v=2 is not caught by this rule. There is one exception: a PDF under /private/press/ can be crawled, because the longer Allow rule wins.

Step 2: remove the lines that do harm

The table below collects, from Google’s documentation, the lines you do not want to find in the file; the “What to do” column is our advice.

Line What happens What to do
Disallow: / (under User-agent: *) The root and every address below it are blocked Remove it if it was left over from a test environment
Disallow: /page/ in the hope that it “removes it from search” Google cannot read the content but may still show the address in results without a snippet Use noindex to keep it out of search
Noindex: /page/ Google does not support a noindex rule inside robots.txt Move the rule to the page’s meta tag or header
Crawl-delay: 10 Google does not support this field It has no effect for Google; remove it if you do not need it
Sitemap: /sitemap.xml Google expects the sitemap to be a full URL, with protocol and host Write https://www.example.com/sitemap.xml
Disallow: /includes/ covering style files According to a comment in Google’s own example file, Google needs these files to render pages As in Google’s example, write an Allow rule for Googlebot, or lift the block

The last row comes from the example file in Google’s documentation: there /includes/ is closed to all crawlers, but because the comment says Google needs those files for rendering, Googlebot is given a separate Allow.

Step 3: publish it and allow for Google’s cache

The file must be called robots.txt and sit at the root of the site. According to Google’s documentation, the file is valid only for the host, protocol and port it is published on: https://example.com/robots.txt does not apply to https://shop.example.com/ or to http://example.com/. A robots.txt in a subfolder is not valid, because crawlers do not look for the file in subfolders.

Once the file is live, you need to know the following:

  • Cache. Google generally caches the contents of robots.txt for up to 24 hours, and may keep it longer when it cannot be refreshed, for example because of timeouts or 5xx errors. A change you make may not reach Google at once.
  • Size. Google limits the file to 500 KiB; content after the limit is ignored.
  • Error cases. On 4xx errors (except 429) Google assumes the file does not exist and that there are no crawl restrictions. On a 5xx error it stops crawling the site for the first 12 hours and uses the last good version for the next 30 days. With redirects it follows at least five hops, then treats the file as a 404.
  • CMS. If you use a service such as Wix or Blogger, you may not be able to edit the file directly; in that case look at the service’s search settings.

If you want separate rules for AI bots, that is covered in its own guide: llms.txt and AI crawlers.

Step 4: prepare the sitemap

First check whether you need one. Google says that if your pages are properly linked, it can usually discover most of your site on its own. It adds that you might need a sitemap if your site is large, if it is new and has few links from outside, or if it has a lot of video and image content. Google defines a “small” site as about 500 pages or fewer. It writes that a site of that size which can be fully crawled by following links from the home page, and has few video or news pages, might not need one. If you use a CMS such as Wix, WordPress or Blogger, the CMS has most likely produced the sitemap already. We cover your site’s link structure in the internal linking guide; a sitemap does not replace it.

Google supports three formats: XML, RSS/Atom and plain text. XML can carry the most information. A basic XML sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/foo.html</loc>
    <lastmod>2026-11-14</lastmod>
  </url>
</urlset>

Decide what goes in using Google’s documentation: the addresses you want to see in search, written as full, absolute URLs. If several addresses lead to the same content, you write the canonical address you prefer, not all of them. A rule of ours: an address that redirects, returns a 404 or carries noindex should not be in the sitemap; the message the sitemap gives and the message the page gives should not contradict each other.

Limits and details:

  • Size. A single sitemap can hold at most 50 MB (uncompressed) and 50,000 URLs. For more, you write several sitemaps and can optionally submit a sitemap index.
  • lastmod. Google writes that it uses the lastmod value if it is consistently and verifiably accurate. The value should reflect the last significant update to the page; a change to the main content, structured data or links counts as significant, while updating a copyright year does not. Stamping today’s date on every address at each release does not meet that test; that reading is ours.
  • priority and changefreq. Google ignores both values.
  • Encoding and location. The file must be UTF-8; Google recommends publishing the sitemap at the root of the site.

If the site has several sitemaps, a sitemap index gathers them in one file:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap-articles.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap-products.xml</loc>
  </sitemap>
</sitemapindex>

Google notes that submitting several sitemaps and index files separately can be useful if you want to track the search performance of each sitemap in Search Console.

Step 5: submit it in Search Console and read the report

Among the ways Google lists are the Sitemaps report in Search Console, the Search Console API and a Sitemap: line in robots.txt. The same document says that submitting is only a hint: it does not guarantee that Google will download the sitemap or crawl its URLs. According to the Search Console Help page, you need owner permission on the property to submit with the report; if you do not have it, you can list the sitemap in robots.txt.

  1. Publish the sitemap on your site and check that the address opens without a login.
  2. In Search Console, run a live URL inspection on the sitemap address; the Help page tells you to check whether Page fetch is “Successful”.
  3. Open the “Sitemaps” report, paste the address into the “Add a new sitemap” box and press “Submit”.
  4. The sitemap is expected to be fetched immediately; crawling the addresses it lists can take time, and not all of them are guaranteed to be crawled.

The statuses in the report are as follows (the “What to do” column is again our advice):

Status What it means What to do
Success The sitemap was fetched and processed without errors; its URLs are queued for crawling Do nothing; compare “Discovered pages” with the number of addresses you expect
Has errors The sitemap was fetched but has at least one error; URLs that parsed without errors are still queued Click the row and read the list of errors
Couldn’t fetch Google could not retrieve the sitemap Check the reasons on the Help page: a robots.txt block, a wrong address (404), a server problem

The report shows only the sitemaps you submitted with the report or the API; sitemaps found through robots.txt are not listed. It shows at most 1,000 submitted requests. To see whether the addresses in a sitemap were indexed, you can filter the Page indexing report by that sitemap; for general use of Search Console, see the Google Search Console guide.

Step 6: test it

These checks take ten minutes after launch:

  1. Open the file in a browser. yourdomain.com/robots.txt should show the content; expect a 200 status code. If you get an error code, how Google behaves is in the list in Step 3.
  2. Paste an important page into URL Inspection. Google’s noindex document tells you to use this tool to check the HTML Googlebot received and whether the noindex rule is visible. On the live test, the Help page looks at the “Crawl allowed?” and “Page fetch” lines under Page availability; the same lines also help you tell whether a page is blocked by robots.txt (that use is ours).
  3. Open the sitemap. Confirm that the addresses are absolute, canonical and open as pages. Checking the status code of every address in the sitemap in bulk with a crawler is our method. Our free SEO checker also looks at a site’s robots.txt and sitemap files.
  4. Look at the robots.txt report in Search Console. It shows when and how Google last read the file; we covered it in step 3 of the technical SEO checklist.
  5. Wait a day. Because of the cache, you may not see the effect of a change at once.

We recorded how a new site shows up in these reports, with a real example, in detmore.tr’s first three weeks.

Common mistakes

  • Thinking Google forgets a sitemap once you delete it. According to the Help page, a sitemap deleted from the report, and the URLs in it, are not forgotten by Google; to stop visits you use a robots.txt rule, delete the file or page, or use noindex.
  • Putting another place’s addresses in the sitemap. The Help page says URLs at a higher level than the sitemap’s directory, or on a different domain, produce a “URL not allowed” error; publish the sitemap at the root of the domain whose addresses it lists.
  • Looking for the sitemap in the wrong property. The report shows only sitemaps in the current property; if you mix up http and https or www and non-www properties, you will not see the sitemap you submitted.

Common questions

What happens if there is no robots.txt file?

According to Google’s documentation, a robots.txt that returns a 4xx (except 429) is treated as if there were no file, and no crawl restrictions are assumed. So the file is not required; you need it only for paths you do not want crawled or for the sitemap address.

Is a sitemap essential for a small site?

Google says a site of about 500 pages or fewer, where every page can be reached through links, might not need one. Even so, submitting costs nothing and the report shows errors in the sitemap; that is our advice.

How do I get a page out of Google?

Make sure the page is not blocked by robots.txt and add noindex to it. According to Google’s documentation, Google has to crawl the page to read the noindex rule, and depending on the page’s importance it can take months for Googlebot to come back; if you are in a hurry, see Google’s removals documentation.

The technical audit in our SEO service covers crawling, indexing and the sitemap; you can see the details on the SEO service page. If you do one thing today, make it this: open yourdomain.com/robots.txt in a browser and check whether there is a Disallow: / line. If there is not, open sitemap.xml in the same window and confirm that the first five addresses in it open.

Sources

This article is general information, not legal, financial or technical advice. Platforms, their rules and the law change over time, so some details may have changed since 14 November 2026 or contain small errors. Before you make a decision for your business, check the current official sources or write to us.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.