Reading time: 11 minutes
The GPTBot user agent and the ClaudeBot user agent collect content that can train models, while OAI-SearchBot and Claude-SearchBot power search. This list of AI bot user agents covers the bots that companies document, that is, the purpose of each one and the robots.txt rule.
In short
- The list comes only from the official pages of 11 companies, so a bot without an official page is left out.
- Each bot has one of three purposes: training, search or user action. At most companies, the search bot decides whether the site shows up in answers, since training has a separate control.
- User action bots, such as ChatGPT-User and Perplexity-User, can ignore robots.txt, according to the companies themselves.
- The list changes, so check the official page before you edit robots.txt.
What is the list of AI bot user agents?
The list of AI bot user agents below comes from the official pages of 11 companies. Each bot has one of three purposes: model training, search or user action. Therefore, copy the names as they are. They are the robots.txt tokens, and the rule for each bot appears next to the table.
Which AI bot user agents are for model training?
Training bots collect pages that can enter model data, that is, what the model learns. According to OpenAI and Anthropic, blocking the GPTBot user agent or the ClaudeBot user agent signals that the content should not train the models.
| Company and user agent | Purpose |
|---|---|
| OpenAI: GPTBot | Model training |
| Anthropic: ClaudeBot | Model training |
| Google: Google-Extended | Gemini training and grounding |
| Apple: Applebot-Extended | Apple model training |
| Meta: meta-externalagent | Content training or indexing |
| Amazon: Amazonbot | Product improvement and training |
| Mistral: MistralAI-Training | Training datasets |
| Common Crawl: CCBot | Open web archive |
According to the companies, every bot in the table has a name for Disallow in robots.txt. Common Crawl, however, only says to block CCBot in the file and does not say in these words that the bot obeys. Two of the names, besides, are not real bots. Google-Extended and Applebot-Extended are robots.txt tokens. In other words, crawling uses the regular bots of each company, and the token only controls the use of what was collected. So neither one shows up in logs. According to Google, Google-Extended has no user agent of its own, and Apple says Applebot-Extended does not crawl pages.
Amazonbot and meta-externalagent mix training and product improvement, according to Amazon and Meta. So decide these two case by case.
Which AI bot user agents are for AI search?
Search bots index pages for answers and citations, and several of them do not train models. Perplexity says this about PerplexityBot, while Amazon, DuckDuckGo and Mistral say the same about Amzn-SearchBot, DuckAssistBot and MistralAI-Index. Therefore, this table holds the access that matters for being cited.
| Company and user agent | Purpose |
|---|---|
| OpenAI: OAI-SearchBot | ChatGPT search |
| Anthropic: Claude-SearchBot | Claude search |
| Perplexity: PerplexityBot | Perplexity search |
| Google: Googlebot | Google Search and its AI features |
| Microsoft: bingbot | Bing and Copilot |
| Apple: Applebot | Siri, Spotlight and Safari |
| Meta: meta-webindexer | Meta AI search |
| Amazon: Amzn-SearchBot | Search in products like Alexa |
| DuckDuckGo: DuckAssistBot | DuckDuckGo AI answers |
| DuckDuckGo: DuckDuckBot | DuckDuckGo search |
| Mistral: MistralAI-Index | Mistral search |
According to the companies, every bot in the table accepts robots.txt rules, but the MistralAI-Index page does not detail this point. Applebot, besides, can also train Apple models, according to Apple itself, unless Applebot-Extended is blocked. In Google, for example, there is no bot only for AI. According to Google’s AI features page, AI is part of Search, and the Googlebot robots.txt rule is the control. Microsoft, for its part, uses the same crawling for Bing and Copilot, according to the Bing guidelines.
Which AI bot user agents open pages at a user’s request?
These bots do not crawl the site on their own. They open a page only when a person asks an assistant a question or a task. Thus, since the request comes from a user, several companies warn that robots.txt may not apply.
| Company and user agent | Purpose | Follows robots.txt? |
|---|---|---|
| OpenAI: ChatGPT-User | Opens pages on request in ChatGPT | May not apply |
| Anthropic: Claude-User | Opens pages on request in Claude | Yes |
| Perplexity: Perplexity-User | Opens pages on request in Perplexity | Usually ignores |
| Google: Google-Agent | Agents that act on user request | Usually ignores |
| Google: Google-GeminiNotebook | Opens sources chosen in Gemini Notebook | Usually ignores |
| Meta: meta-externalfetcher | Opens links on user request | May ignore |
| Amazon: Amzn-User | Answers Alexa queries | May ignore some |
| Mistral: MistralAI-User | Opens pages on request in Vibe | Not detailed by the company |
Anthropic is the exception because, according to the company, its bots respect robots.txt. In the other cases, the rule varies. Google, for example, explains that a user-triggered fetcher usually ignores the rules, and Mistral does not detail the point. So, to really block these bots, put the rule in the firewall.
Which other AI user agents show up in logs?
Four bots from the list do not fit the three purposes, but they show up in logs. GoogleOther is Google’s generic crawler, used, for example, in research and development. Google-CloudVertexBot crawls only at the request of site owners, because it helps build agents in Vertex AI.
OAI-AdsBot and meta-externalads, on the other hand, are tied to ads. The first one validates pages submitted as ads in ChatGPT, but OpenAI says it does not train models. The second one crawls the web to improve advertising products.

How do AI bot user agents show up in logs?
In logs, the GPTBot user agent shows up as a long line, that is, with the bot name, the version and a contact address. Filter by the name, not by the whole string, because the version changes. The examples below come from the official pages, so copy the bot name from there.
GPTBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Googlebot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
bingbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
Amazonbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Amzn-SearchBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
MistralAI-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
CCBot CCBot/2.0 (https://commoncrawl.org/faq/)
DuckAssistBot DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
meta-webindexer meta-webindexer/1.1
Three details prevent mistakes when you filter:
- Anthropic: the company publishes only the names ClaudeBot, Claude-SearchBot and Claude-User, without the full string. So you find the ClaudeBot user agent only by its name.
- Wildcard: Google, Microsoft and Amazon use Chrome/W.X.Y.Z in place of the version, so use a wildcard in that part.
- robots.txt mark: whenever it fetches the file, OpenAI may put “robots.txt” in the user agent string of GPTBot and OAI-SearchBot.
The commands to count requests per bot are, by the way, in the guide to AI crawlers in server logs.
How do you allow or block AI user agents in robots.txt?
Write one User-agent group for each bot you want to treat differently, starting from the list of AI bot user agents above. The companies say each setting is independent of the others. Thus, to refuse training, write one group for the GPTBot user agent and another for the ClaudeBot user agent, and leave OAI-SearchBot free.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Six rules from the official pages change the result:
- No group of its own: Amzn-SearchBot follows the rules of the other search bots if the file names them, and Applebot follows the rules of Googlebot if the file names Googlebot.
- Each subdomain: Anthropic asks that the block go in the robots.txt of every subdomain, that is, one file per host.
- Crawl-delay: Anthropic accepts the directive, but Applebot and the Amazon bots do not.
- Time frame: OpenAI and Amazon say about 24 hours for the change to apply, while Meta says up to 24 hours and DuckDuckGo, for DuckAssistBot, says 72.
- IP blocking: Anthropic warns that it can fail, because it stops the bot from reading robots.txt.
- User action: since robots.txt may not apply, the block needs a firewall or CDN.
The decision for each bot, however, depends on your goal, and the effects for GEO are in the guide to robots.txt for AI. After you edit the file, follow the walkthrough to test whether robots.txt blocks Google, Bing or AI bots. If the bot is blocked outside the file, that is, at the CDN, see how to allow AI search bots on Cloudflare. For a complete file, with search allowed and training refused, copy the robots.txt template for AI.
How do you verify that the user agents are real?
To check the GPTBot user agent, the ClaudeBot user agent and the others, compare the request IP with the official lists. After all, anyone can write a fake user agent. Google itself warns that the string can be spoofed, and Common Crawl warns about bots that pretend to be CCBot. Besides, Google, Apple and Common Crawl describe reverse DNS on their pages.
- OpenAI: a JSON list of IPs for each bot, including GPTBot, OAI-SearchBot and ChatGPT-User.
- Anthropic: a single list, bots.json, for the three bots.
- Perplexity: one list for PerplexityBot and another for Perplexity-User.
- Google: common-crawlers.json and reverse DNS ending in googlebot.com. The user-triggered fetchers have their own lists.
- Microsoft: the Verify Bingbot tool in Bing Webmaster Tools.
- Apple: reverse DNS at applebot.apple.com or a list of CIDR blocks.
- Amazon: a list of IPs for Amazonbot, Amzn-SearchBot and Amzn-User.
- DuckDuckGo, Mistral and Common Crawl: JSON lists of IPs. At Mistral, they cover MistralAI-User and MistralAI-Index. Common Crawl also accepts reverse DNS at crawl.commoncrawl.org.
- Meta: the page for its AI bots does not publish an IP list.
Finally, the script that matches the log IP against these lists is in the same logs guide.
How often does the list of user agents change?
The list of user agents changes without a fixed calendar, because each company edits its own page. On Google’s page, for example, Google-NotebookLM shows up as a former agent, supported until August 2026, and Google-GeminiNotebook takes its place. In 2026, Meta updated its page in May, Apple published its page in September, and Google and Amazon edited theirs in October.
OpenAI, for example, warns that the GPTBot version can change, and Anthropic publishes only the ClaudeBot name. Cohere, on the other hand, says it does not use bots to train models today and would list a bot in its table if that changes. Besides, Anthropic offers a form to get a notice when its article changes in a relevant way.
Therefore, treat the list of user agents as a snapshot. Review it every quarter or after each launch of a new assistant. Always prefer the official pages, because they change first.
Frequently asked questions
The GPTBot user agent collects content that can train OpenAI models, while OAI-SearchBot shows sites in ChatGPT search. The robots.txt rules of each one are independent. So blocking only GPTBot refuses training without taking the site out of search, as the guide to robots.txt for AI shows.
Not necessarily. According to Anthropic, the ClaudeBot user agent collects content that can contribute to training, while Claude search uses Claude-SearchBot. Disabling Claude-SearchBot, however, can reduce the visibility and accuracy of the site in Claude search results.
No. Google-Extended has no user agent of its own. Google crawls with its usual user agents, and the token only controls the use of the content in Gemini training and grounding. Therefore, robots.txt is the only place where it shows up.
First, check whether it is a user action bot, because the companies say robots.txt may not apply to these bots. Next, confirm the IP against the official lists. To block it, use the firewall or the CDN, as in the guide on ChatGPT blocked on Cloudflare.
Main sources
- OpenAI, Overview of OpenAI Crawlers (2026).
- Anthropic, Anthropic’s bots and how to block them (2026).
- Google Search Central, list of Google’s common crawlers (2026).
- Google Search Central, list of user-triggered fetchers (2026).
- Perplexity, Perplexity Crawlers (2026).
- Apple, About Applebot (2026).
- Amazon, Amazon Web Search crawlers (2026).
- Meta, Meta Web Crawlers (2026).



