llms.txt and AI crawlers: how to set up robots.txt

From GPTBot to PerplexityBot: what each bot does, what blocking it costs, and the steps to write and test robots.txt and llms.txt to match your own decision.

· 15 min readSEO

Short answer

AI bots come in three groups: training, search and user-triggered fetchers. To appear in AI answers, keep the search bots allowed in robots.txt; if you do not want training, close only the training bots. llms.txt does not manage access: it is a proposal that summarises the site, and it does not replace robots.txt.

A manufacturer asks ChatGPT about suppliers in its own sector. The answer names two competitors and links to their sites; its own is missing. The cause is not always content: three lines copied from a blog post two years ago, or a single setting left on in the CDN panel, may be stopping AI bots from reading the site. The subject of llms.txt and AI crawlers comes down to two files: robots.txt, which says which bot may read the site, and llms.txt, which guides whoever reads it. In this guide you will get to know the bots, make your decision, set up robots.txt to match it and test the result.

We covered crawling and indexing checks on the Google side in the technical SEO checklist. Here we look only at AI systems.

Two files, two separate jobs

  • robots.txt sits at the root of the site and tells crawlers which addresses they may crawl. Rules are written per bot name (user-agent). It is not a lock; following the rule is up to whoever runs the bot. The companies in this article state in their own documentation that their bots can be managed with robots.txt.
  • llms.txt holds no access rules. It is a file that summarises what the site is and its important pages in Markdown, and guides the system reading it.

For content that really must stay private, neither is enough; that content belongs behind a login.

AI bots fall into three groups

People talk about “the AI bot” as if it were one thing, yet there are three groups doing three different jobs. Training bots collect content that may be used to train models. Search bots crawl the sites to be shown in AI search answers. User-triggered fetchers do not roam on their own; when someone asks an assistant a question, they open the relevant page at that moment. Which one you close decides what you lose.

The three groups of AI bots, the bots in each group and the cost of blocking Group Bots Cost of blocking Training bots GPTBot, ClaudeBot, Google-Extended, CCBot Content is not used to train models Search bots OAI-SearchBot, PerplexityBot, Claude-SearchBot You do not appear in AI search answers User-triggered fetchers ChatGPT-User, Perplexity-User, Claude-User The page the user asked for may not open The three groups of AI bots, the bots in each group and the cost of blocking Group: Training bots Bots GPTBot, ClaudeBot, Google-Extended, CCBot Cost of blocking Content is not used to train models Group: Search bots Bots OAI-SearchBot, PerplexityBot, Claude-SearchBot Cost of blocking You do not appear in AI search answers Group: User-triggered fetchers Bots ChatGPT-User, Perplexity-User, Claude-User Cost of blocking The page the user asked for may not open
Figure: AI bots fall into three groups, and blocking each group costs something different.

The table below sums up the ten names you will meet most often, each as its operator’s own documentation describes it.

Bot Operator What it does If you block it
GPTBot OpenAI Crawls content that may be used to train generative AI models You signal that your content should not be used in training
ClaudeBot Anthropic Collects web content that could contribute to model training The site’s future content is kept out of training datasets
Google-Extended Google Not a separate bot; it is a control name used in robots.txt. It manages the use of content for training Gemini models and for grounding answers in Gemini apps Use on the Gemini side is switched off; your inclusion and ranking in Google Search are not affected
CCBot Common Crawl Crawls for an open web crawl archive that anyone can access and analyse Your site is not taken into this archive
OAI-SearchBot OpenAI Crawls sites to show them in ChatGPT’s search features You are not shown in ChatGPT search answers; you may still appear only as a navigational link
PerplexityBot Perplexity Crawls sites to show and link them in Perplexity search results; it does not collect content for foundation models Your appearance in Perplexity results is put at risk
Claude-SearchBot Anthropic Indexes content to improve the quality of search results Your visibility in search and the accuracy of answers about you may fall
ChatGPT-User OpenAI Opens the page when a user asks ChatGPT or a custom GPT a question; it does not crawl automatically OpenAI writes that robots.txt rules may not apply, because a user starts these requests
Perplexity-User Perplexity Visits the page while answering a user’s question According to Perplexity, this bot generally ignores robots.txt rules
Claude-User Anthropic Accesses the site when a user asks Claude a question Claude cannot retrieve your content for the user’s question

We count CCBot in the training group because the archive is open to everyone: AI developers can draw on it too. Google’s AI features inside Search are not in this table, because no separate bot crawls for them. According to Google’s documentation, AI is part of Search and is controlled with Googlebot rules.

Step 1: See where you stand today

  1. Open yoursite.com/robots.txt in a browser. If you have subdomains (shop, blog, support), look at each one’s file separately; Anthropic’s documentation also asks for the block to be written for every subdomain.
  2. Search the file for the ten names in the table and for any Disallow: / line standing alone.
  3. Note who added each rule, when and why. Often the answer is “we don’t know”, which shows the decision was never made.

Step 2: Make the decision

The decision is commercial more than technical, and it comes down to three options.

Your goal Training bots Search bots User-triggered fetchers
To appear in AI answers Open Open Open
Not to be used for training Closed Up to you Up to you
Both: to appear, but not be used for training Closed Open Open

The third row is possible because the companies keep these settings independent of one another. OpenAI’s documentation gives the example itself: a site owner can allow OAI-SearchBot in order to appear in search results while closing GPTBot. The one cost is on the Google side: closing Google-Extended also switches off the use of your content for grounding answers in Gemini apps.

For a business that sells services and wants to be known, the first or third row usually makes sense. For publishers whose income comes from the content itself the sums are different, and the decision is made with legal advice and the revenue model considered together.

Step 3: Write robots.txt

If you are not going to block any bot, you do not need to write their names; a bot that is not named follows the general group (User-agent: *):

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://www.example.com/sitemap.xml

If you are going to name the bots, there is one rule you need to know: a crawler follows the single group in the file that names it most specifically, and ignores the other groups. Google’s robots.txt documentation says so plainly. So in every group you name, you must repeat the restrictions from the general group as well. The example below applies the third row of the decision table:

User-agent: *
Allow: /
Disallow: /admin/

# AI search and user-triggered fetchers: open
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

# Model training: closed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Once the file is live, be patient. OpenAI writes that for search results it can take about 24 hours for its systems to adjust to a robots.txt change, and Perplexity that it may take up to 24 hours for a change to be reflected.

There is also a shared signal on the way. Google has noted in its documentation that it intends to support a shared training preference signal, based on the AIPREF work at the IETF, in early 2027. Anthropic has announced that as of 30 October 2026 it has begun recognising a “no AI training” preference, and that a robots.txt rule naming ClaudeBot takes priority over this general signal. For today, rules written per bot name are the route that works.

Step 4: Check that the bots can actually reach you

robots.txt may be saying “come in” while the firewall shuts the door. Check in order:

  1. Read the live file. Some CDNs can add their own rules to the top of robots.txt. Do not trust the file in the repository; open the live address in a browser.
  2. Look at the CDN and firewall settings. Bot protection, ready-made options of the “block AI bots” kind, and country or rate-limit rules can stop these bots. Perplexity’s documentation warns that on Cloudflare, while Bot Fight Mode is on, its bots can still be challenged even if a custom allow rule has been written.
  3. Send a request under the bot’s name. On the command line, type curl -I -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yoursite.com/. If a 403 or a verification page comes back instead of 200, a rule that works on the name is blocking the bot. This test checks only the name; the request leaves from your IP address, so it does not show rules that work on IP.
  4. Look at the server or CDN logs. Filter the requests that arrived under these names in recent weeks and the status codes they received. If there are no requests at all, or every one is a 403, the problem lies somewhere before robots.txt.
  5. Write the allow rule with the IP lists. OpenAI, Perplexity and Anthropic publish the IP ranges of their bots. OpenAI and Perplexity recommend allowing requests from these ranges if you want to appear.

When you want to block, the route is robots.txt too. Anthropic notes that blocking IPs prevents the bot from reading robots.txt, so it is not reliable as a lasting opt-out.

Step 5: Add llms.txt

llms.txt is a proposal published by Jeremy Howard on 3 September 2024 and kept at llmstxt.org. The idea is this: web pages are made for people; pulling clean text out from among menus, scripts and decoration is hard work for a language model. An /llms.txt file at the root of the site gives a short introduction and links to the pages where the detail lives. According to the proposal the file is used most in software documentation; the same structure also works for a business that wants to outline its structure and policies.

The format is defined and its order is fixed:

  1. An H1 heading with the name of the site or project. This is the only required section.
  2. A blockquote line with a short summary.
  3. Optional paragraphs or lists of detail.
  4. Link lists under H2 headings: one link per line, followed if you wish by a colon and a short note.
  5. The section named “Optional” is for secondary links that can be skipped when a shorter context is needed.

An example in its smallest form:

# Example Furniture

> Example Furniture is a workshop that makes desks and cabinets to measure for offices. Turkish pages are at the root, English pages under /en/.

- How we work: discovery, measuring, written proposal

## Services

- [Office desks](https://www.example.com/en/office-desks/): made to measure, delivered and assembled
- [Common questions](https://www.example.com/en/faq/): delivery, warranty and care

## Optional

- [Blog](https://www.example.com/en/blog/)

You need to know its limits as well:

  • It is a proposal; it has not been adopted as a standard. No system is obliged to read it. None of the OpenAI, Anthropic and Perplexity crawler documents we read for this article lists llms.txt as the way to manage their bots; all of them point to robots.txt.
  • Google Search does not need it. Google’s AI features documentation says you do not need to create new machine-readable files or AI text files to appear in AI Overviews and AI Mode.
  • It is of more use at answer time than in training. The proposal expects the file to be used when an agent needs information while helping a user.
  • It does not manage access. It does not stop a bot and does not hide a page.

Even so, it is easy to add and it does one job: you describe your business in your own sentence to an agent that arrives at your site. The proposal’s site also notes that Chrome’s Lighthouse tool looks for this file in its agentic browsing audits. The proposal describes the test itself: give an assistant only your llms.txt file and ask it questions about your business.

How we set it up at Detmore

On our own site we chose the first row of the decision table. detmore.tr/robots.txt writes the general group, the search bots, the user-triggered fetchers and the training bots by name as separate groups; the site is open to all of them, only /api/ is closed, and that restriction is repeated in every group. Our reasoning is simple: being known to AI assistants helps an agency be recommended. Because our site is served through Cloudflare, we keep the managed robots.txt and the option to block AI bots switched off in the panel; if they were on, they would override this file.

detmore.tr/llms.txt is not written by hand. It is regenerated on every deploy from the same data that produces the pages: a one-sentence definition, contact details and how we work, then the services and guides with a one-sentence summary each, in Turkish and English. When a new article is published it enters the file by itself. Our web and software team includes this kind of work in the site build.

Common mistakes

  • Closing everything with the wildcard. Disallow: / under User-agent: * does not stop only AI bots; it stops Googlebot too. This line is usually left over from the test environment; it is one of the launch-day checks in keeping SEO through a site rebuild.
  • Acting as if there were a single “AI bot”. Closing GPTBot does not take you out of ChatGPT search; closing OAI-SearchBot does. The cost changes with the group.
  • Closing Google-Extended and expecting to leave Google’s AI summaries. The AI features inside Search depend on Googlebot rules. To limit the information shown in Search, Google points to the nosnippet, data-nosnippet, max-snippet and noindex controls.
  • Confusing noindex with Disallow. Disallow stops crawling; noindex stops indexing. A bot cannot read the noindex tag on a page you have closed in robots.txt. If you want a page out of search, leave crawling open and use noindex.
  • Not repeating the restrictions in a named group. A GPTBot group that says only Allow: / cancels the general group’s prohibitions for that bot.
  • A stale llms.txt. Links to removed pages and services you no longer offer lead the assistant to describe you wrongly. Either generate the file automatically or update it with every new page and every service you drop.
  • Forgetting the block in the CDN. While robots.txt is open, a bot rule sitting in the panel quietly returns a 403 and nobody notices.

Bots being able to read the site only opens the door. A place in the answers is still the content’s job: pages that answer the question directly and make clear who wrote them. We covered the search side of this in B2B SEO.

Common questions

Does adding llms.txt affect Google rankings?

According to Google’s documentation, you do not need to create AI text files to appear in the AI features in Search. Do not treat llms.txt as a ranking tool; add it as a helper file that describes your site to an agent by the short route.

If I block GPTBot, will I still appear in ChatGPT?

According to OpenAI’s documentation, yes: GPTBot is for training, OAI-SearchBot manages visibility in ChatGPT search, and the two settings are independent of each other. You can allow OAI-SearchBot and close GPTBot.

Can a bot I have blocked in robots.txt still open my page?

For bots that crawl on their own, the companies write that they follow the rule. With user-triggered fetchers it is different: OpenAI says the rules may not apply to ChatGPT-User, and Perplexity that Perplexity-User generally ignores them. Content that nobody should read belongs behind a login.

Here is something you can do in ten minutes today: open yoursite.com/robots.txt, search for the ten names in the table one by one and write “did we ask for this?” beside every rule you find. Then run the command in Step 4 once. If either surprises you, go back to the decision table. If you would like the rest of the file and crawling as a whole looked at together, this check is part of the technical audit in the first month of our SEO service.

Sources

This article is general information, not legal, financial or technical advice. Platforms, their rules and the law change over time, so some details may have changed since 10 November 2026 or contain small errors. Before you make a decision for your business, check the current official sources or write to us.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.