AI Crawlers and robots.txt: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot

"Should we block AI bots?" has no single yes or no answer, because there is no single bot. The same company visits your content with one crawler to train models, another to show pages in search results, and a third to carry out a request from a user. A site that shuts them all out with one line gets a result it did not want: it tries to opt out of training and drops out of search answers as well.

This guide sorts the crawlers by the job they do and gives ready-made robots.txt examples for three common decisions. Bot names and behaviour are taken from the documentation published by OpenAI, Anthropic, Perplexity, Google and Microsoft (read on 12 October 2026). Vendors change these lists; check the current page in the sources before you decide.

Three types of bot

The bots, vendor by vendor

OpenAI: GPTBot vs OAI-SearchBot

This is the pair people confuse most. According to OpenAI's crawler documentation:

• GPTBot crawls content that may be used to train generative models. Blocking it signals that you do not want your content used for training.
• OAI-SearchBot is for search. Sites that block this bot are not shown in ChatGPT's search answers; they can still appear as navigational links only.
• The two settings are independent. You can disallow GPTBot and leave OAI-SearchBot allowed, and the documentation describes this explicitly.
• ChatGPT-User does not crawl automatically; it visits a page when a user asks for something in ChatGPT or in a custom GPT. Because the user initiates these requests, robots.txt rules may not apply.
• A robots.txt change can take about 24 hours to be reflected on the search side.

The ChatGPT-specific steps are in how to appear in ChatGPT.

Anthropic: three Claude bots

Anthropic's support page describes three bots: ClaudeBot collects content that could contribute to training; Claude-SearchBot builds an index to improve the quality of search results; Claude-User goes to a page when a user asks a question. According to the page, blocking Claude-SearchBot or Claude-User may reduce your site's visibility in search results and in user-directed searches.

Anthropic states that its bots honour the standard directives in robots.txt and support the non-standard Crawl-delay extension. It also notes that blocking IP addresses is not a reliable method, and that the preference should be declared in robots.txt, separately for each subdomain.

In older robots.txt files you will come across an anthropic-ai line. That name is not listed on Anthropic's current page; the current names are the three above.

Perplexity: two bots

According to Perplexity's documentation, PerplexityBot exists to surface and link websites in Perplexity's search results; it does not crawl content to train foundation models. Perplexity-User visits a page when a user asks a question, and the documentation says plainly that, because the user initiates the request, this bot generally ignores robots.txt rules. Changes can take up to 24 hours to be reflected.

Google: there is no separate "AI bot"

Google's AI Overviews and AI Mode work from the normal search index. So the crawler that matters is Googlebot; a site that blocks it drops out of both classic results and AI answers.

Google-Extended is not a bot but a control token used in robots.txt. According to Google's crawler list, it governs whether your content may be used to train future Gemini models and for grounding in Gemini apps; it does not affect a site's inclusion in Google Search and is not used as a ranking signal. In other words, blocking Google-Extended does not remove you from AI Overviews. A separate setting in Search Console is used for that; the details are in Google AI Overviews optimization.

Microsoft: Bingbot and Copilot

The Bing Webmaster Guidelines cover Bing search, Copilot and grounding results together. Here control works through meta tags more than robots.txt: according to the guidelines, noindex removes the page from Bing search, Copilot and grounding; noarchive prevents the content from being used in Copilot answers and grounding results; and nocache limits what Copilot can use to the URL, title and snippet.

Ready-made robots.txt for three decisions

The examples below are starting points. Add the relevant blocks without deleting the other rules already in your file.

A. Appear in search, opt out of training

This is the setup people ask for most often.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

B. Allow everything

You do not need to write a special rule; with no blocking line, the bots are treated as allowed. Still, check that no old User-agent: * / Disallow: / has been left in the file.

C. Block them all

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /

With this setup you are not cited as a source in ChatGPT, Claude and Perplexity search answers. Because some user-triggered bots (ChatGPT-User, Perplexity-User) are not bound by robots.txt, a site that wants a complete block also has to write rules at the server or firewall level.

robots.txt is not the only gate: CDN and firewall

robots.txt is a request; what actually stops a bot or lets it through is the layer in front of your server. Problems arise in both directions:

• You think you are open, but you are closed. CDN and security services offer one-click settings along the lines of "block AI bots". If that setting is on, the bot gets a 403 even though robots.txt allows it. This is why OpenAI and Perplexity recommend also allowing the published IP ranges of their search bots.
• You think you are closed, but you are open. A user-triggered bot you have disallowed in robots.txt may still arrive. A firm block needs a rule based on user agent and IP range.

To verify, search your server access logs for the bot names and look at the status code returned: 200 means the bot gets through, 403 means it is blocked. Because a user agent name can be faked, compare suspicious requests against the IP list the vendor publishes (OpenAI, Anthropic and Perplexity publish these lists as JSON).

What is llms.txt, and what is it not?

llms.txt is a proposal for a file placed at the root of a site that tells language models, in Markdown, what the site is and which pages matter most. It was proposed by Jeremy Howard in September 2024; in the site's own words it is an informal specification that is open to community input.

Three things need to be kept apart:

• It is not robots.txt. It does not grant or remove access. To block a bot you use robots.txt, not llms.txt.
• Google Search does not use it. Google's guide to generative AI search says so explicitly: having the file neither harms nor helps your visibility on Google.
• There is no evidence for the other vendors. The crawler documentation from OpenAI, Anthropic and Perplexity contains no statement that llms.txt affects whether you appear in answers.

It costs little to prepare and does no harm; the proposal expects the file to be useful when an AI tool reads the site at that moment, rather than for model training. But "we added llms.txt, so AI will see us now" is an assumption, not a result.

Checklist

• A separate decision has been made for training, search and user-triggered bots
• There is no forgotten blanket block under User-agent: * in robots.txt
• The search bots of the platforms I want to appear on are allowed (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot)
• The AI bot setting in my CDN and firewall matches my decision
• Server logs show these bots receiving a 200
• The robots.txt file of each subdomain (blog., shop.) has been checked
• At least 24 hours have passed since the change
• The list is compared with the vendor documentation every three months

Instead of checking the first four items by hand, you can use the free AI bot and llms.txt checker: you enter your domain, and the tool shows in a table what your robots.txt says to each bot and checks the format of your llms.txt file. The tool reads the rules; only your server logs can show whether the bots actually visit. SEOLOK's AI visibility tracking module also looks at your robots.txt at the start of each measurement run and, if AI search bots are blocked, reports it in the dashboard and with a warning.

Frequently asked questions

Does blocking GPTBot hurt my SEO?

It has no effect on your Google rankings; GPTBot is OpenAI's training bot. It does not affect your visibility in ChatGPT search either; OAI-SearchBot determines that.

If I allow the bots, will I appear in AI answers?

Permission is a precondition, not a result. Even with access open, whether you appear in an answer depends on your content and on the question itself. To see whether you appear, read the measurement guide.

Will the bots put a strain on my server?

For a bot that crawls heavily, use the method its vendor supports. Anthropic states that it supports the Crawl-delay directive. Where another vendor's documentation has no such statement, set the rate limit on the server or CDN side.

Can I close off only certain folders?

Yes. Path-based rules such as Disallow: /account/ apply to these bots as well. Closing account, basket and internal search result pages while leaving content pages open is a common setup.

Access is the first step. What to do once the gate is open is the subject of the what is GEO guide.