Reading time: 12 minutes
robots.txt for AI is the set of rules that tells each AI crawler what it may crawl. To show up in answers, allow the search crawlers in robots.txt, such as OAI-SearchBot for ChatGPT, Claude-SearchBot and PerplexityBot. Training crawlers, such as GPTBot and ClaudeBot, can stay blocked in robots.txt without removing your site from those answers.
In short
- At most AI companies, robots.txt controls search and training separately. So the search crawler decides whether your site shows up in answers.
- Blocking GPTBot in robots.txt does not remove your site from ChatGPT search. But blocking OAI-SearchBot in robots.txt removes it from ChatGPT search answers, according to OpenAI.
- Blocking Google-Extended in robots.txt does not remove your site from AI Overviews. That is because it limits something else: Gemini training and grounding in the Gemini apps.
- robots.txt is a request, not a lock. Your host or CDN can block a crawler that the file allows, so test each user agent.
Which AI crawlers exist and what does each one do?
AI crawlers do three jobs. They find pages to cite in answers, collect content to train models and open pages when a user asks. OpenAI and Anthropic, for example, have one crawler for each job. Google and Apple, however, control training through a separate token in robots.txt, without a crawler of their own.
| Company | Search crawler | Training crawler or token |
|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot |
| Anthropic | Claude-SearchBot | ClaudeBot |
| Perplexity | PerplexityBot | None declared |
| Googlebot | Google-Extended (token) | |
| Apple | Applebot (Siri, Spotlight and Safari) | Applebot-Extended (token) |
| Amazon | Amzn-SearchBot (Alexa) | Amazonbot (mixed use) |
| Meta | No search-only crawler | meta-externalagent (mixed use) |
| Common Crawl | No search | CCBot (open archive) |
| Microsoft | Bingbot (Bing and Copilot) | No training token |
There are also crawlers that open pages when a user asks: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher and Amzn-User. Be careful with them, because not all of them follow robots.txt. According to Perplexity’s documentation, Perplexity-User generally ignores the file, since a person asked for the visit. Similarly, OpenAI says robots.txt rules may not apply to ChatGPT-User. Meta and Amazon give the same warning about their crawlers. Anthropic, on the other hand, says its crawlers, including Claude-User, follow robots.txt.
How do Apple and Microsoft handle robots.txt?
There’s a catch with Apple. According to Applebot’s documentation, if robots.txt doesn’t mention Applebot but mentions Googlebot, Apple’s crawler follows the Googlebot rules. So a block meant for Google can also apply to Siri and Spotlight search.
Currently, Microsoft has no AI token in robots.txt. According to Cloudflare, Microsoft plans to support training opt-outs through the file in early 2027. Meanwhile, Bingbot crawls for both Bing and Copilot, and Bing’s guidelines ask you not to block it. To keep a page out of Copilot answers, Bing points to the NOARCHIVE meta tag.
How many sites already have AI crawler rules in robots.txt?
Few sites write rules for these crawlers. In the HTTP Archive’s Web Almanac 2025, GPTBot appeared in the robots.txt of 4.5% of desktop sites in July 2025. In 2024, the figure was 2.9%. However, naming a crawler doesn’t mean blocking it, because the rule may allow the site or restrict just one folder.
Almost no one blocks search. According to Cloudflare, which sells bot controls, less than 1% of sites on its network block search crawlers. By contrast, 17% use some kind of training block. The Web Almanac 2025 sums up the file’s new role:
“robots.txt has become a dual-purpose control point, balancing visibility in search with protection against large-scale data harvesting.”
SEO chapter of the Web Almanac 2025, HTTP Archive (2026)
Does blocking GPTBot remove your site from ChatGPT?
No. GPTBot collects content to train OpenAI’s models, while OAI-SearchBot decides whether your site shows up in ChatGPT search. According to OpenAI, a site that blocks OAI-SearchBot in robots.txt drops out of ChatGPT search answers. Even so, it can still appear as a navigational link.
In OpenAI’s crawler documentation, the rules for each crawler in robots.txt are independent. This means you can allow OAI-SearchBot and block GPTBot in the same robots.txt. If you allow both, OpenAI may use a single crawl for both purposes, which avoids repeat visits.
Blocking GPTBot does have a cost for GEO, though. Without search, ChatGPT answers with what the model learned in training. With GPTBot blocked in robots.txt, the site signals that its content should not go into the training of future models. So, in those models’ memory, your brand comes to depend on what other sites say about it. The difference between memory and search is in the guide to what GEO is.
ChatGPT-User is a different case. It opens pages when a ChatGPT user asks, and OpenAI says robots.txt may not apply to those visits. The ChatGPT-User rule in robots.txt also doesn’t decide who appears in search. The role of each crawler, and of Bing, is in the guide on how to show up in ChatGPT.
How do you allow search and block AI training in robots.txt?
In robots.txt, block only the training crawlers and tokens, such as GPTBot, and let search crawlers follow the general rules. This works because a crawler without its own group obeys the User-agent: * group. As a result, if that group doesn’t block the site, robots.txt already allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, with no extra lines.
For example, a store that wants to appear in AI answers can block GPTBot and other training crawlers with this robots.txt:
# Model training: refused
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Everyone else, including the AI search crawlers
User-agent: *
Disallow: /cart/
Disallow: /my-account/
Sitemap: https://www.yourstore.com/sitemap.xmlCCBot builds Common Crawl’s open archive, which anyone can access. Cloudflare even includes it in the list of training crawlers in the robots.txt it generates. Amazonbot and meta-externalagent also collect content for training, according to Amazon and Meta, but they serve other products at those companies. Decide on those two case by case.
Each crawler follows only one group: the one with its name or, if there is none, the * group. Under Google’s specification, the crawler ignores the other groups. Therefore, if robots.txt gets a group for OAI-SearchBot with Allow: /, that crawler stops obeying the * rules, such as the cart block. In that case, repeat in the new group the rules that should still apply.
We left Google-Extended out of the example on purpose. Once you block it, you opt out of Gemini training, but you also remove your content from grounding answers in the Gemini apps. Anyone who wants to appear as a source in the Gemini app should therefore keep it allowed.
Why doesn’t robots.txt stop an AI crawler on its own?
The robots.txt file is a request, not a lock. According to Cloudflare, following the file is voluntary, and it doesn’t prevent access at a technical level. To enforce a block, then, the rule goes into the CDN or firewall. On Cloudflare, for example, the Disallow option for training, with Bot Preference Sync turned on, writes the rules into robots.txt. It also blocks training-only crawlers at the edge. The step-by-step is in the guide on how to check and allow AI crawlers on Cloudflare.
Does Google-Extended affect AI Overviews?
No. AI Overviews and AI Mode are part of Google Search, so they follow the Googlebot rules. Google-Extended controls something else: the use of content to train Gemini models and to ground answers in the Gemini apps. Therefore, blocking it in robots.txt doesn’t remove your site from Search or from AI Overviews.
According to Google’s documentation, Google-Extended doesn’t affect a site’s inclusion in Search, and it isn’t a ranking signal. It also has no user agent of its own: Google’s regular crawlers do the crawling, and the token only signals the permitted use. That is why it doesn’t show up in your logs.
There are two ways to take your site out of AI Overviews and AI Mode. The first is the Search Console generative AI control, which Google rolled out to all sites on August 31, 2026. The second is on-page display controls, such as nosnippet, because these features only use pages that can show a snippet.
| Control | What it removes | What it doesn’t remove |
|---|---|---|
| Google-Extended | Gemini training and grounding in the apps | Search, AI Overviews and AI Mode |
| Search Console control | AI Overviews, AI Mode and AI in Discover | The rest of Search and training |
| nosnippet | Page snippets, including in AI Overviews | The Search index |
According to Search Console Help, the control doesn’t affect AI training. To limit training of the models that generate those answers, Google points to Google-Extended. On the other hand, noindex removes the page from Search entirely. Control details, such as inheritance between properties, are in the guide to Google’s AI Mode and AI Overviews.
How do you test whether robots.txt allows AI crawlers?
Test three layers: the file, the rules and the server response. First, open the robots.txt on each host. Then find out which group applies to each crawler and see how your host responds to each user agent. After all, a correct file isn’t enough if the server or CDN blocks the crawler first.
- Open the file on each host. The address
https://www.yoursite.com/robots.txtmust return a 200, as plain text. Each subdomain and each protocol has its own file, and one doesn’t inherit the other’s rules. - Find each crawler’s group. Look for the name of each search and training crawler. Without its own group, the
*group applies. Then check that noDisallow: /catches a search crawler. - Use the free reports. The Search Console robots.txt report shows the files Google found, when it last read them and any errors. The Bing Webmaster Tools tester shows which line blocks a URL, but only for Bing’s crawlers.
- Check your logs. Filter by crawler and count the response codes. When it fetches the file, OpenAI may add a “robots.txt” marker to the user agent. Since anyone can fake a user agent, check the IP against the lists the companies publish.
- Test the server with each user agent. The block may sit with your host or CDN, outside robots.txt and the CMS. The script below requests the file and the home page with each crawler’s name.
SITE="https://www.yoursite.com"
for BOT in OAI-SearchBot ChatGPT-User GPTBot Claude-SearchBot ClaudeBot PerplexityBot; do
UA="Mozilla/5.0 (compatible; $BOT)"
R=$(curl -sL -o /dev/null -w "%{http_code}" -A "$UA" "$SITE/robots.txt")
H=$(curl -sL -o /dev/null -w "%{http_code}" -A "$UA" "$SITE/")
echo "$BOT | robots.txt: $R | home page: $H"
doneWhen is the block outside robots.txt?
At NotHuman, our blog’s robots.txt doesn’t block GPTBot, Google-Extended or CCBot, as a GEO decision. Even so, in our own test, we sent a request from a regular IP with GPTBot’s full user agent. The hosting server responded with a 429, including on robots.txt. Requests with the OAI-SearchBot, ChatGPT-User, ClaudeBot and browser user agents got a 200. We don’t know whether the real crawler, coming from OpenAI’s IPs, also gets a 429, because the test doesn’t reproduce the IP. The lesson is to test each user agent and check the logs, since the block can sit with the host, outside robots.txt and WordPress.
If a crawler gets a 403, 429 or 503 while a browser gets a 200, the block comes from the server or the CDN. The script above, however, uses only each crawler’s name. For the full user agent, and to see the text the crawler receives, use the test in the guide to JavaScript SEO for AI crawlers.
A server error when delivering the file also blocks crawlers. Under RFC 9309, the robots.txt standard, a 5xx error requires the crawler to assume everything is disallowed. After you fix robots.txt, OpenAI says ChatGPT search can take about 24 hours to adjust. Finally, robots.txt is only the first of the reasons AI doesn’t cite a brand. After it come the CDN, the index and the content.
Frequently asked questions
Not reliably. The robots.txt file is a request that the big companies’ crawlers say they follow, not a technical lock. Some crawlers that open pages at a user’s request, such as Perplexity-User, generally ignore the file. So to actually block access, the rule has to live in the server, the CDN or the firewall.
About 24 hours for the crawlers that state a time frame. OpenAI and Amazon say about 24 hours, while Perplexity says up to 24 hours. Google usually caches robots.txt for up to 24 hours. Finally, the crawler still has to come back to your pages.
Not necessarily. According to OpenAI, ChatGPT-User opens pages at users’ request, and robots.txt rules may not apply to those visits. ChatGPT-User also doesn’t decide whether your site shows up in ChatGPT search. For search, the OAI-SearchBot rule in robots.txt is what counts.
No. According to the llms.txt proposal, the two files have different purposes. The robots.txt file says what access is acceptable, while llms.txt offers a site summary for AI to consult. Google also says you don’t need to create AI text files to appear in AI Overviews and AI Mode. See what llms.txt is for and whether it’s worth creating one.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- OpenAI, Overview of OpenAI Crawlers (2026).
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (2026).
- Perplexity, Perplexity Crawlers (2026).
- Google Search Central, Google’s common crawlers and the Google-Extended token (2026).
- Search Console Help, Search generative AI control (2026).
- Google Search Central, How Google interprets the robots.txt specification (2026).
- IETF, RFC 9309, Robots Exclusion Protocol (2022).
- HTTP Archive, Web Almanac 2025, SEO chapter (2026).
