Reading time: 12 minutes
Cloudflare AI Crawl Control is the dashboard that shows which AI crawlers access your site and lets you allow or block each one. It works alongside Cloudflare’s AI policies, which since July 2026 have separated search, agent and training. So, for your site to show up in ChatGPT, a Cloudflare block must not reach search.
In short
- Since September 15, 2026, Cloudflare’s presets for new domains have kept search allowed. For sites with ads, however, the preset disallows training.
- Anyone who had the old “Block AI bots” turned on and never used the three policies moved to “Disallow AI Training”. This option blocks GPTBot and ClaudeBot but keeps Googlebot and Bingbot. Cloudflare also now blocks agents, such as ChatGPT-User, on pages with ads.
- Choosing “Block” or “Block on pages with ads” for training now also blocks Googlebot, Bingbot and Applebot, with an effect on traditional search.
- Setting “Allow” in AI Crawl Control does not create an allow rule. So other WAF rules and the Cloudflare AI policies can still block OAI-SearchBot, the crawler behind ChatGPT search.
Does Cloudflare block AI crawlers by default?
Today, not for search. Since September 15, 2026, both presets offered to new domains leave search allowed. In the preset for sites without ads, the Cloudflare AI policies start with no block on any crawlers. The preset for sites with ads disallows training and blocks agents on pages with ads, but it keeps search. Existing domains moved based on their previous choices.
Even so, a block at the CDN is one of the most common reasons AI does not cite a brand, and Cloudflare is the first place to check.
Since when does Cloudflare block AI crawlers?
Cloudflare’s reputation for blocking dates back to 2025. On July 1 of that year, Cloudflare announced that it would block AI crawlers without permission or payment by default. After that, every new domain had to answer, at sign-up, whether it wanted to allow these crawlers.
On July 1, 2026, the “Your site, your rules” post replaced the single toggle with three policies. Search covers crawlers that index content to answer questions later. Agent covers crawlers that act in real time on a person’s behalf, such as ChatGPT-User. Finally, Training covers crawlers that train models. According to the documentation, the old “Block AI bots” blocked training crawlers but left out mixed-use ones, which do both search and training.
What did Cloudflare change for AI crawlers on September 15?
On September 15, the “Have it both ways” post set out what applies today. The table sums up the presets for new domains, as well as the migration for those who never touched the three policies:
| Domain status | Training | Agent |
|---|---|---|
| New, without ads | Allow | Allow |
| New, with ads | Disallow AI Training | Block on pages with ads |
| Existing, with “Block AI bots” on | Disallow AI Training | Block on pages with ads |
| Existing, without “Block AI bots” | Allow | Allow |
In other words, Search stays on Allow in all four cases. Those who had already configured the three policies kept the practical effect, and Block or Block on pages with ads for training became Disallow AI Training. You can also change the presets for new domains at sign-up or later.
According to the same post, less than 1% of Cloudflare sites block search crawlers, and 17% turn on some kind of training block. However, the figure comes from Cloudflare itself, which sells bot control services. Note: the Block AI Bots page in the documentation, updated on July 1, still describes the plan from before September 15.
Which Cloudflare controls can block ChatGPT?
Three places in the Cloudflare dashboard decide whether to block OpenAI’s AI crawlers: the AI policies, the Crawlers tab in AI Crawl Control and the WAF custom rules. In addition, each crawler falls under a different policy: OAI-SearchBot is search, GPTBot is training and ChatGPT-User is agent. Bot Fight Mode is also worth checking.
In the Cloudflare Radar bots directory, OAI-SearchBot appears in the Search category, GPTBot in Training and ChatGPT-User in Agent. However, the Category column in AI Crawl Control uses the old names AI Search, AI Crawler and AI Assistant, as Cloudflare’s bot reference shows.
| Control | Where it is | What it blocks |
|---|---|---|
| Search | Configure AI bot policies | Search crawlers, such as OAI-SearchBot |
| Training | Configure AI bot policies | GPTBot and ClaudeBot; with Block, also Googlebot, Bingbot and Applebot |
| Agent | Configure AI bot policies | Crawlers acting at a user’s request, such as ChatGPT-User |
| Action | AI Crawl Control, Crawlers tab | The chosen crawler, through a WAF rule |
| Custom rules | Security rules | Whatever the rule defines |
Each one carries a different weight: according to OpenAI’s documentation, sites that block OAI-SearchBot do not appear in ChatGPT search answers. On the other hand, blocking GPTBot only opts the site out of training. Blocking Bingbot on Cloudflare also matters, because Bing feeds ChatGPT, as the guide on how to show up in ChatGPT shows.
Bot Fight Mode challenges traffic with bot patterns and does not accept exceptions through WAF rules. When it acts, the event shows up with Bot Fight Mode as the service. The search and training crawlers of the other engines are in the table in the guide on what GEO is.
How can you see in the logs whether GPTBot and OAI-SearchBot were blocked?
Start with what anyone can see: the public robots.txt. Then open the Crawlers tab in AI Crawl Control, the status chart in the Metrics tab and the security events. Together, they show whether the crawler arrived, whether it got a block code and which feature blocked it.
- Open the site’s robots.txt. Look for the “BEGIN Cloudflare Managed content” or “BEGIN Cloudflare Bot Preference Sync” blocks. A “Disallow: /” under GPTBot opts the site out of training. Under OAI-SearchBot, however, it removes the site from ChatGPT search.
- Check the Crawlers tab. In AI Crawl Control, find GPTBot and OAI-SearchBot. The Action column shows Allow or Block.
- Filter the Metrics tab by crawler. The Status code distribution chart separates 2xx from 4xx responses. The 4xx group includes, for example, the 403 for blocks and the 402 for charges.
- Open the security events. On the Analytics page, Events tab, look for the crawler among the user agents in Top events by source and then select Filter. The Events by service section shows which feature acted, such as a custom rule or Bot Fight Mode.
- Check the rule order. In Security rules, filter by Custom rules and find the rule named “AI Crawl Control”. However, a rule above it can block a crawler you allowed there.
What else should you check in Cloudflare AI crawler data?
To begin with, the managed robots.txt deserves attention for Gemini. That is because the example in Cloudflare’s robots.txt documentation includes “Disallow: /” for Google-Extended. According to Google, this token controls grounding in Gemini Apps, as the guide on how to show up in Gemini, Perplexity and Claude explains.
In addition, the documentation gives four caveats for reading this data:
- On the free plan, AI Crawl Control recognizes crawlers only by user agent, and the Metrics tab covers the last 24 hours.
- Likewise, on the free plan, security events show only sampled logs.
- The Requests column adds up allowed and unsuccessful requests. An unsuccessful request can come from any rule or from a server error.
- After you change robots.txt, the Directives tab may flag as violations past requests that the file allowed at the time.
How do you allow search crawlers and keep training blocked?
To stay in ChatGPT, leave Search on Allow and, under Training, choose the Disallow option, which Cloudflare calls Disallow AI Training. This lets Cloudflare block AI crawlers used only for training, such as GPTBot and ClaudeBot, but keep Googlebot, Bingbot and Applebot in search. Then check in Cloudflare AI Crawl Control that the search crawlers show Allow.
The steps below follow the robots.txt documentation. You can switch the dashboard to other languages, but the names below are the ones in the documentation, in English.
- On the Security Settings page, filter by Bot traffic and then select Configure AI bot policies.
- Under Search, choose Allow.
- Under Training, choose Disallow. Avoid Block and Block on pages with ads, because they now also block Googlebot, Bingbot and Applebot.
- Under Agent, decide with the client. With Block on pages with ads, Cloudflare keeps ChatGPT-User out of pages with ads.
- Turn on Enable Bot Preference Sync and select Save. The robots.txt then reflects the policies.
- Finally, in the Crawlers tab in AI Crawl Control, leave OAI-SearchBot, PerplexityBot and Claude-SearchBot on Allow.
The September 15 post is direct about Block for training, which now reaches search too:
“It will stop Applebot, Bingbot, and Googlebot from reaching your site…”
Bryan Becker, Cloudflare (2026)
Why doesn’t Allow undo every Cloudflare block?
Setting Allow in Cloudflare AI Crawl Control does not create an allow rule. According to the documentation on AI Crawl Control with the WAF, its rule only includes blocked crawlers. So a crawler allowed there can still hit a rule that runs earlier, and that rule may not show up in AI Crawl Control analytics. In that case, adjust the rule that blocks the crawler.
Allow also does not undo a block in the AI policies. After all, in the order described by the same documentation, Cloudflare’s bot solutions act after the AI Crawl Control rule. So check both screens.
There is, however, an exception with Bing. That is because Microsoft expects to support training opt-outs through robots.txt in early 2027. Until then, Disallow AI Training does not pass that opt-out on to Bing, according to the same September 15 post. To decide crawler by crawler, see which crawlers to allow and which to block in robots.txt.
Does Cloudflare AI Crawl Control charge AI crawlers for access?
Not by default. AI Crawl Control is available on all Cloudflare plans, including the free one. However, charging is a separate feature, pay per crawl, in closed beta. With it, the site owner sets a price. Then a crawler that does not present payment intent gets a 402 code instead of the page.
According to the pay per crawl documentation, Cloudflare acts as the payment intermediary. The price is the same for every crawler on Charge, and files such as robots.txt and sitemap.xml are always free. Blocks from the WAF and Bot Management take precedence over charging.
For GEO, charging a search crawler has the same practical effect as blocking it until it pays. So read the code carefully in the logs. On paid plans, the block response can be a 403 or a 402, with custom text, according to the Manage AI crawlers page. In other words, a 402 does not always mean charging is active.
How long does it take for AI to cite the site again after you allow it?
There is no guaranteed timeline. OpenAI and Perplexity mention about 24 hours for their systems to reflect a change in the rules. After that, the crawler needs to crawl the pages again, and citation still depends on the question and the content.
In OpenAI’s documentation, the timeline is about 24 hours after the robots.txt update, for search results. Similarly, in Perplexity’s documentation, changes can take up to 24 hours. However, both timelines refer to robots.txt rules. To track the recovery, follow three steps:
- Confirm access. In the Status code distribution chart, OAI-SearchBot requests should get 2xx again.
- Check Bing. If Bingbot hit a block, resubmit the sitemap and the main pages in Bing Webmaster Tools. The step-by-step is in the guide on how to index your site on Bing.
- Measure with fixed prompts. Compare weeks, not individual answers, because the answer changes with each run.
The measurement protocol covers the number of runs and the margin of error. It is in the guide on how to measure your brand’s mentions and citations in AI. Finally, visits that come back through ChatGPT show up in AI traffic in GA4.
What we found in Portuguese-language pages about Cloudflare and AI crawlers
We read the 7 Portuguese-language pages that came up in our searches about the Cloudflare block on AI crawlers. All are news stories: two from 2024, four from July 2025 and one from July 2026. None mentions OAI-SearchBot or AI Crawl Control.
One mentions AI Audit, the dashboard’s old name. In addition, none shows how to check whether a block affects the site, and none mentions Googlebot. Finally, the only one from 2026 describes the July plan, from before the September 15 changes.
Frequently asked questions
No. According to OpenAI, blocking GPTBot only opts the site out of training. What removes the site from ChatGPT search answers is blocking OAI-SearchBot, which Cloudflare classifies as a search crawler. Blocking Bingbot on Cloudflare also matters, because Bing feeds ChatGPT.
Not yet. According to Cloudflare’s September 15, 2026 post, Microsoft expects to support training opt-outs through robots.txt in early 2027. Until then, Disallow AI Training does not pass that opt-out on to Bing.
The 402 code is pay per crawl’s response to a crawler that does not present payment intent. But a 402 does not always mean charging is active. On paid plans, the block response that Cloudflare AI Crawl Control sends to crawlers can be a 403 or a 402, with custom text.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training, September 15, 2026.
- Cloudflare, Your site, your rules: new AI traffic options for all customers, July 1, 2026.
- Cloudflare Docs, AI Crawl Control with Cloudflare WAF (2026).
- Cloudflare Docs, robots.txt setting and Bot Preference Sync (2026).
- OpenAI, Overview of OpenAI Crawlers (2026).
