robots.txt template for AI: copy, paste and adjust

Three ready robots.txt blocks to copy, with what each one does: allow the AI bots, block only training and block the listed AI bots. It includes the bot names from OpenAI, Anthropic, Google, Perplexity and Microsoft, common errors and a test routine.
Written with AI, reviewed by humans

In this article

See how your brand shows up in AI answers

Book a demo and find out what to fix first, with a GEO specialist.

Book a demo

Reading time: 11 minutes

A robots.txt template for AI needs one block for each decision: allow everything, block only training or block the AI bots. To show up in answers, allow the search bots, but treat the training bots as the site’s choice. Below are three ready blocks, with what each one does and how to test it.

In short

  • Allowing each company’s search bot is the first step to show up in AI answers, but it does not guarantee a citation. According to OpenAI, blocking GPTBot, the training bot, does not remove your site from ChatGPT search.
  • The most balanced robots.txt template for AI allows search and helps you block AI bots that only train. So GPTBot, ClaudeBot and Google-Extended go under Disallow, and the rest follows the * group.
  • A bot with its own group ignores the * group, according to Google. So repeat the rules that still apply in the new group.
  • After you edit the file, test the rules, the server response and the logs, because OpenAI says search takes about 24 hours to adjust.

What is the difference between search and training bots in robots.txt for AI?

A search bot crawls pages to cite them in answers, while a training bot collects content to train models. OpenAI, Anthropic and Google separate the two in robots.txt, so you can allow one and block the other. User bots, however, open a page only when someone asks, and not all of them follow the file.

CompanySearch botTraining bot
OpenAIOAI-SearchBotGPTBot
AnthropicClaude-SearchBotClaudeBot
PerplexityPerplexityBotNone declared
GoogleGooglebotGoogle-Extended (token)
MicrosoftBingbotNone documented

According to OpenAI, OAI-SearchBot shows sites in ChatGPT search, while GPTBot collects content that may train the models. Each rule works on its own. Likewise, Anthropic separates the two: ClaudeBot collects content that may enter training, and Claude-SearchBot improves the quality of search results. Perplexity, on the other hand, says PerplexityBot does not crawl content to train foundation models.

At Google, Googlebot feeds Search, but Google-Extended is only a token. So it controls the training of future Gemini generations and grounding in the Gemini apps, without affecting the site’s inclusion in Search. According to Bing’s guidelines, Bing and Copilot use the same crawling foundation, and Microsoft asks you not to block Bingbot in robots.txt. The full theory on each bot, however, is in the guide on which AI bots to allow and which to block.

What about the bots that act on a user’s request?

ChatGPT-User, Claude-User, Perplexity-User and Google-Agent open pages when a user asks, so they do not crawl the site on their own. According to OpenAI, robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores the file. Anthropic, on the other hand, says its bots respect robots.txt. Likewise, Google’s user-triggered fetchers usually ignore the file. The full list, with the user agent and the IP of each bot, is in the list of AI bot user agents.

robots.txt template for AI: illustration of open gates that AI bots pass through and one closed gate that stops a bot

Which robots.txt template for AI allows all bots?

To allow all AI bots, just do not block them, because without a block rule access is open. Even so, an explicit block is worth it: it records the decision and keeps private folders closed. So copy Template 1 and replace /cart/ and /my-account/, the cart and account folders of a typical store, with the private folders of your site.

# Template 1: allows search and training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /cart/
Disallow: /my-account/
Allow: /

# All others, including Googlebot and Bingbot
User-agent: *
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://www.yoursite.com/sitemap.xml

So the group with the names puts the search, user and training bots under the same rules. Googlebot and Bingbot stay out, because they follow the * group. Each bot follows only one group: according to Google’s specification, the most specific group applies, and the * group does not add to it. So Template 1 repeats the private folders in the group of the AI bots. The Sitemap line, by the way, belongs to no group and applies to all bots.

This robots.txt template for AI works for any type of site. If the site has no private folders, then delete the Disallow lines and keep only Allow: /. Allowing only lets bots read the pages and does not guarantee a citation.

How do you block only AI training in robots.txt?

To block the training bots, put GPTBot, ClaudeBot and Google-Extended under Disallow: / and leave the search bots in the * group. So OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot stay allowed without any line of their own. This is the scenario OpenAI describes, that is, allow OAI-SearchBot to appear in search and block GPTBot.

# Template 2: allows search and refuses training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# All others, including the AI search bots
User-agent: *
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://www.yoursite.com/sitemap.xml

This robots.txt template for AI leaves search open and closes training from now on. However, four details change the result:

  • Google-Extended: blocking it refuses the training of future Gemini generations, but it also takes your content out of grounding in the Gemini apps. Also, it does not affect Search. So if you want Gemini to be able to use your content as a source, remove this line from the block.
  • Perplexity and Microsoft: there is no training bot to block. Perplexity says PerplexityBot does not crawl content for foundation models, and Bing documents no training bot.
  • Reach of the block: it applies from now on. Anthropic talks about future content, and none of the companies’ pages promises to remove what was already collected.
  • Cost for GEO: with GPTBot and ClaudeBot blocked, the site signals that the content should not enter training. So the guide on the cost of blocking training shows what this changes for the brand.

How do you block AI bots in robots.txt?

To block the AI bots in this guide, repeat Disallow: / for each one. The price is leaving the answers: according to OpenAI, a site that blocks OAI-SearchBot does not show up in ChatGPT search answers, but it may appear as a navigational link. Therefore, use this robots.txt for AI bots only if the decision is not to be cited.

# Template 3: blocks the AI bots in this guide
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Disallow: /

# All others, including Googlebot and Bingbot
User-agent: *
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://www.yoursite.com/sitemap.xml

Template 3 covers only the listed bots, and others, such as CCBot and Applebot-Extended, need lines of their own. It also leaves Googlebot and Bingbot out on purpose, because both feed traditional search. According to Google, the Googlebot directive is the crawl control for Search, that is, blocking it affects all of Search. To limit only what Search shows, Google points to nosnippet, data-nosnippet, max-snippet or noindex. On Bing, however, the NOARCHIVE meta tag prevents the use of content in Copilot answers.

Finally, robots.txt is a request, not a lock, since RFC 9309 says the protocol is not a substitute for security measures. So to really block the AI bots, use the firewall or the CDN. On Cloudflare, for example, the guide on how to allow AI search bots without opening the site to scrapers shows how to keep training blocked.

How do you test a robots.txt template for AI after editing?

Test the robots.txt template for AI in three steps: the rules, the server response and the logs, because each one catches a different error. First, check in a local script which bots the file allows. Then publish the file and test the server with the user agent of each bot.

To test the rules before publishing, save the template as robots.txt and run the script below in the same folder. It uses only Python’s standard library, so it shows, bot by bot, whether the test page is allowed. The script prints allowed or BLOCKED for each bot.

from urllib.robotparser import RobotFileParser

PAGE = "https://www.yoursite.com/product/example"
BOTS = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot",
        "Claude-User", "ClaudeBot", "PerplexityBot", "Perplexity-User",
        "Google-Extended", "Googlebot", "Bingbot"]

rules = RobotFileParser()
with open("robots.txt", encoding="utf-8") as robots_file:
    rules.parse(robots_file.read().splitlines())

for bot in BOTS:
    status = "allowed" if rules.can_fetch(bot, PAGE) else "BLOCKED"
    print(f"{bot:18} {status}")

The script, however, shows only what the file says and does not see the CDN, the firewall or the host. Also, up to Python 3.12 the library applies the first rule that matches, not the longest one. So put the specific rule before the general one, as in the templates. So after you publish, run three checks:

  1. Publishing: upload the file to the root of each host, because it applies only to the protocol, port and host it was read from. Also, Anthropic asks for the block on each subdomain.
  2. Response: open the file in the browser, because it must return 200, as plain text with UTF-8 encoding.
  3. Server: test each user agent, because the right file is not enough if the host or the CDN blocks the bot. So the guide on how to test robots.txt on Google, Bing and AI has the curl command and how to read each code.

What common errors break crawling?

The most common errors close the whole site, such as a Disallow: / in the * group or a server error response. Others leave out only the AI bots, such as a new group that wipes the * rules. The six items below cover each case.

  • Disallow: / in the * group: the slash alone blocks the site for every bot without its own group, for example when a staging file goes live.
  • Named group that wipes the *: a bot that gets its own group stops following the *. So repeat the private folders in it, as in Template 1.
  • Wrong bot name: the group applies only if the name is the one the company publishes, such as OAI-SearchBot. With another name, the bot falls into the * group.
  • Nonstandard file: according to Google, the file must be UTF-8, and Google ignores content beyond 500 KiB. Also, if HTML comes back instead, Google tries to extract the rules and discards the rest.
  • Server error: according to RFC 9309, a 5xx error requires the bot to assume a full block. Likewise, Google stops crawling the site during the first 12 hours of 5xx errors. A 429, unlike other 4xx codes, does not count as a missing file for Google.
  • IP or firewall block: Anthropic warns that blocking the IPs of its bots can stop it from reading robots.txt. So the bot does not see even the rule that allows it. On Cloudflare, for example, the guide on how to check and allow AI bots shows where to look.

How do you confirm in the logs that the bots came back?

Wait about 24 hours, because OpenAI and Perplexity say a change takes up to that long. Then filter the log by the name of each bot and check two things: whether it read /robots.txt and which code it got on the pages. So a 200 on both shows that the bot came back and got the page.

LOG=access.log   # compressed log (.gz): swap grep for zgrep

for BOT in OAI-SearchBot GPTBot Claude-SearchBot ClaudeBot PerplexityBot; do
  echo "== $BOT (requests per day, type and code)"
  grep -i "$BOT" "$LOG" \
    | awk '{ kind = ($7 == "/robots.txt") ? "robots.txt" : "page"; print substr($4, 2, 11), kind, $9 }' \
    | sort | uniq -c
done

The command reads the combined format of Apache and Nginx, so adjust the fields if your log is different. The header says requests per day, type and code, and the type is robots.txt or page. For example, a bot with 403 or 429 only on the pages points to a block outside robots.txt. If it does not show up, the file may be wrong or the bot has not come back yet.

Anyone can copy a user agent, so confirm the IP. OpenAI publishes IP lists in JSON for OAI-SearchBot and GPTBot, as Perplexity does for its bots. Also, when it fetches robots.txt, OpenAI may add the “robots.txt” marker to the user agent, which helps separate those requests. Google-Extended does not appear in the logs, because it has no user agent of its own. Finally, the IP check commands are in the guide on how to tell in the logs whether ChatGPT, Perplexity and Googlebot visit the site.

Frequently asked questions

Do I need to list Googlebot and Bingbot in robots.txt for AI?

No, because without a group of their own, both follow the * group and stay allowed if it does not block the site. Googlebot controls Search crawling, including the AI features, and Bing’s guidelines ask you not to block Bingbot. Therefore, create a group for them only if you need a different rule.

Can I allow GPTBot only on the blog and block the rest?

Yes, because under RFC 9309 the most specific rule wins, that is, the one with the longest path. With Allow: /blog/ and Disallow: / in the GPTBot group, the bot reads only the blog. Even so, run the test script before publishing, to check pages inside and outside the folder.

Does blocking Google-Extended hurt my ranking on Google?

No, according to Google, because Google-Extended does not affect the site’s inclusion in Search or work as a ranking signal. It controls only Gemini training and grounding in the Gemini apps. Search and its AI features follow Googlebot.

Want to know how your brand shows up in AI answers today?

Book a demo and see how your brand shows up in AI answers, with support from a GEO specialist.

Main sources

About this articleWritten with AI and reviewed by humans, with verified sources. Found something out of date? Write to the NotHuman team.

Get new NotHuman articles by email

By subscribing, you agree to receive blog articles by email and to the Privacy Policy. Unsubscribe any time.

Keep reading

Demo

Want to know how your brand shows up in AI answers today?

Book a demo and see how your brand shows up in AI answers, with support from a GEO specialist.

Book a demo

Your privacy

We use necessary cookies to make the site work and Google Analytics and Microsoft Clarity statistics cookies to improve our content. You can decline the statistics ones; in the EU, UK and Switzerland they are only turned on after you accept. Cookie Policy.

Cookie preferences

Cookie Policy