Reading time: 11 minutes
If Cloudflare is blocking ChatGPT, fix it in the dashboard. First, set Search and Agent to Allow in the AI policies. That lets in OAI-SearchBot and ChatGPT-User. Then set Training to Disallow with Bot Preference Sync, which blocks GPTBot and adds the refusal to the Cloudflare robots.txt. Finally, review the Cloudflare WAF and Bot Fight Mode.
In short
- On Cloudflare, OAI-SearchBot falls under the Search policy, ChatGPT-User under Agent, and GPTBot under Training. So you can keep search and block training, with the refusal in the Cloudflare robots.txt.
- A Skip rule in the Cloudflare WAF lets verified search bots in. However, it can’t cover every verified bot, because GPTBot is verified too.
- Bot Fight Mode on the free plan doesn’t accept rule-based exceptions. So if it blocks a search bot, turn it off or allow the bot’s IPs.
- PerplexityBot is not on Cloudflare’s list of verified bots. Therefore, allowing it depends on the IP ranges Perplexity publishes.
Does Cloudflare block AI bots by default?
Not for search. Since September 15, 2026, the presets for new domains set Search to Allow. So when Cloudflare is blocking ChatGPT, the cause may be an old choice or a custom rule. For example, sites that had the old Block AI bots setting on, and never used the three policies, started blocking ChatGPT-User on pages with ads.
The diagnosis is in the guide on how to check whether Cloudflare blocks AI crawlers. The final setup looks like this:
| Bot | Cloudflare policy | Setting |
|---|---|---|
| OAI-SearchBot and Claude-SearchBot | Search | Allow |
| ChatGPT-User and Claude-User | Agent | Allow |
| GPTBot and ClaudeBot | Training | Disallow |
| PerplexityBot and Perplexity-User | Not on the verified list | IP rule in the WAF |
The categories come from Cloudflare Radar. Meanwhile, the role of each bot is in the guide to what GEO is.
How do you allow search bots on Cloudflare and block training bots?
The setup has four parts: AI policies, the AI Crawl Control bot list, WAF rules, and bot protection. Cloudflare WAF custom rules run before the bot solutions. As a result, an old rule can keep Cloudflare blocking ChatGPT even with the policies set to Allow.
- AI policies: in Security Settings, filter by Bot traffic and open Configure AI bot policies. Set Search and Agent to Allow, and choose Disallow for Training. Then turn on Bot Preference Sync, which adds the refusal to the robots.txt that Cloudflare serves. Avoid Block for training, because since September 15, 2026 it also blocks Googlebot, Bingbot, and Applebot.
- AI Crawl Control: in the bot list, set search bots and agents to Allow. Use Block only for bots the training policy doesn’t cover.
- Exception rule: in the Cloudflare WAF, create the Skip rule from the next section, right below the AI Crawl Control rule.
- Bot protection: on the free plan, check whether Bot Fight Mode acted on a search bot. On paid plans, however, set Verified bots to Allow in Super Bot Fight Mode.
How do you create the Cloudflare WAF rule that allows ChatGPT?
If Cloudflare is blocking ChatGPT because of an old rule, an exception fixes it without turning protection off. In Security rules, create a custom rule with the Skip action. The condition combines the Known Bots field, which flags bots verified by Cloudflare, with each bot’s name in the user agent. Verification uses IP, reverse DNS, or a signature. So we infer that a scraper that only copies the name stays outside the exception:
cf.client.bot and (
http.user_agent contains "OAI-SearchBot" or
http.user_agent contains "ChatGPT-User" or
http.user_agent contains "Claude-SearchBot" or
http.user_agent contains "Claude-User"
)Next, choose what to skip from the Skip options, such as the remaining custom rules and Super Bot Fight Mode. Also keep Log matching requests on, which logs each allowed request in the events.
Watch out for two mistakes:
- Using Known Bots alone: GPTBot and ClaudeBot are verified too. So a rule above the AI Crawl Control block that skips the remaining rules lets both through.
- Forgetting the limit: the free plan has 5 custom rules, and an AI Crawl Control block takes one of them.
Why does PerplexityBot need an IP rule?
Because PerplexityBot and Perplexity-User are not in the Cloudflare Radar directory of verified bots. In August 2025, Cloudflare removed Perplexity from the list. It accused Perplexity of crawling sites without identifying itself. The same day, Perplexity disputed it. It said Cloudflare had confused its traffic with a third-party cloud browser service. Since Known Bots only flags verified bots, we infer that it doesn’t apply to Perplexity’s bots.
The Perplexity documentation shows the rule for the Cloudflare WAF: user agent PerplexityBot or Perplexity-User, and source IP in the published ranges. However, it uses the Allow action from the old firewall rules. In custom rules, use Skip instead and keep the ranges in an IP list, which the free plan also has:
(http.user_agent contains "PerplexityBot" or
http.user_agent contains "Perplexity-User")
and ip.src in $ips_perplexityHow do you keep scrapers and training bots blocked?
With Disallow for training and Bot Preference Sync on, Cloudflare blocks training bots at the edge. Mixed-use bots, however, such as Googlebot and Bingbot, stay allowed. Against scrapers that don’t identify themselves, there are two more layers:
- Bot protection: Bot Fight Mode or, on paid plans, Definitely automated set to Block in Super Bot Fight Mode.
- AI Labyrinth: available on all plans, it traps AI bots that ignore the rules in a maze of links. According to Cloudflare, it doesn’t affect SEO.
What changes with agents that sign their requests?
ChatGPT’s cloud browser (ChatGPT agent, on Cloudflare) proves who it is with a signature, Web Bot Auth. According to OpenAI, its requests carry the Signature-Agent header “https://chatgpt.com”, and Cloudflare checks the signature on its own.
Since July 1, 2026, Cloudflare has treated these agents as verified bots, with the intermediary label. Because ChatGPT agent is in the Agent category, we infer that the Agent policy also applies to it. OpenAI also shows a rule based on Bot Detection ID 129220581. However, that field requires Bot Management on the Enterprise plan.
Does Bot Fight Mode block ChatGPT and other AI bots?
The documentation doesn’t say it targets AI bots. Bot Fight Mode challenges traffic that matches known bot patterns. Historically, though, verified bots have been excluded from default settings. Still, this free-plan feature accepts no exceptions, not even a Cloudflare WAF rule.
So if Cloudflare is blocking ChatGPT through this feature, the security events show Bot Fight Mode in the Service field. In that case, there are three documented ways out:
- Turn the feature off in Security Settings.
- Create IP Access rules with the Allow action for the bot’s IPs. After all, Bot Fight Mode doesn’t act when one of these rules matches the request first.
- Move to the Pro plan, which includes Super Bot Fight Mode, with Skip rules and the option to set Verified bots to Allow.
However, according to the IP Access rules documentation, Allow skips custom rules, rate limiting, and managed rules. It also doesn’t show up in the events. In addition, a rule only accepts a single IP or /24 and /16 ranges, but the OpenAI and Perplexity lists have smaller ranges.
Does the Cloudflare managed robots.txt replace mine?
No. If the site already has a robots.txt, Cloudflare puts the managed block before the file’s content and serves both in a single response. On the other hand, with no file, Cloudflare creates a robots.txt that blocks known AI bots. However, the block only appears with Bot Preference Sync on and at least one restrictive policy.
In the public file, check three things:
- Google-Extended: the documentation example has a Disallow for this token. According to Google, it also controls grounding of answers in Gemini apps.
- Content signals: the block includes Content Signals, such as search=yes and ai-train=no. Cloudflare also warns that Search Console sometimes flags these robots.txt lines as unrecognized syntax, with no observed effect.
- A request, not a lock: because robots.txt is voluntary, Cloudflare blocks at the edge with Disallow for training or Block in AI Crawl Control.
To decide bot by bot, see which bots to allow and which to block in robots.txt.
How do you confirm Cloudflare is not blocking ChatGPT?
Confirm it on three dashboard screens, then from outside. Trace simulates the request, the events show which rules acted, and Security Analytics shows whether the request reached the server. Finally, an outside test reveals blocks at the host.
- Trace: simulate a GET with the OAI-SearchBot user agent. The result lists the rules in the order they run, but it doesn’t show IP Access rules. Without the bot’s IP, the Known Bots part of the Skip rule may not match there.
- Security events: on the Analytics page, Events tab, filter by the bot’s user agent. With logging on, the Cloudflare WAF Skip rule appears on each allowed request.
- Security Analytics: check whether the bot’s requests show as Served by origin, that is, whether they reached the server.
- Outside test: request the page with each bot’s user agent, as in the curl test from the JavaScript SEO guide. The test doesn’t reproduce the bot’s IP, but it shows user agent blocks.
If Cloudflare keeps blocking ChatGPT after these changes, the Service field in the events shows which feature acted. On Cloudflare, most security blocks return 403, while rate limiting returns 429. But a 429 can also come from the host, as the next section shows.
Finally, give it time: OpenAI mentions about 24 hours for search to reflect a robots.txt change.
Do other firewalls and CDNs do the same?
Yes. Any firewall, CDN, or host can block an AI bot before it even reads robots.txt. That’s why AI companies give guidance on exceptions. OpenAI recommends allowing the OAI-SearchBot IP ranges, and Perplexity shows the rule for AWS WAF. In any dashboard, a safe exception combines user agent and IP, as Perplexity recommends.
There are also these options:
- Anthropic: a single IP list. But Anthropic warns that blocking IPs can prevent it from reading robots.txt.
- Google: reverse DNS or the crawler IP lists, according to the Google documentation.
- Signed agents: OpenAI provides guidance for ChatGPT’s browser on Akamai, Vercel, and HUMAN.
If Cloudflare isn’t blocking ChatGPT and the bot still fails, look at the host. The server’s firewall can also block a bot, and Perplexity warns that any WAF may need to allow its bots explicitly. So test each user agent and check the logs, as the guide on when the block is outside robots.txt shows.
After all, the CDN is just one of the technical layers that the SEO and AI guide covers. The full CDN check is part of the technical SEO audit for the AI era.
Frequently asked questions
Yes. The free plan has the AI policies, AI Crawl Control, up to 5 custom rules in the Cloudflare WAF, and IP Access rules. It doesn’t have Super Bot Fight Mode. So if Bot Fight Mode blocks a search bot, the way out is to turn it off or allow the bot’s IPs.
It can copy the name, but not the verification. The Known Bots rule only applies to bots that Cloudflare confirmed by IP, reverse DNS, or signature. So a scraper that uses the ChatGPT-User user agent is still subject to the site’s other protections.
It depends on the rule. With the Known Bots field, Cloudflare itself keeps the bot verification up to date. On the other hand, the Perplexity IP rule depends on current ranges, and Perplexity recommends automating updates from its official lists.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- Cloudflare Docs, Verified bots (2026).
- Cloudflare Docs, Available skip options (2026).
- Cloudflare Docs, Bot Fight Mode (2026).
- Cloudflare Docs, robots.txt setting and Bot Preference Sync (2026).
- OpenAI, Overview of OpenAI Crawlers (2026).
- OpenAI, ChatGPT Work’s Cloud browser allowlisting (2026).
- Perplexity, Perplexity Crawlers and WAF configuration (2026).
- Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives, August 4, 2025.
