Reading time: 11 minutes
ChatGPT shows up in logs with the OAI-SearchBot user agent when it crawls pages for search, or ChatGPT-User when it opens them for users. GPTBot, also from OpenAI, collects content for training. In your server logs, the user agent also identifies Googlebot, Bingbot and the AI crawlers from Claude and Perplexity.
In short
- In ChatGPT, the OAI-SearchBot user agent means a search crawl, while ChatGPT-User means a user request. GPTBot, on the other hand, collects content for training.
- Anyone can copy the Googlebot or ChatGPT user agent. So check the IP against each company’s official lists.
- Logs live on the server, not in WordPress. On hosts with cPanel, for example, Raw Access gives you the access log.
- Logs show the visit, but they don’t say if the page became a citation. In fact, Google-Extended never shows up, because it has no user agent of its own.
What user agent does ChatGPT use?
OpenAI uses one user agent for each task, and two of them serve ChatGPT answers. OAI-SearchBot crawls pages for ChatGPT search, and ChatGPT-User opens a page when a user asks. GPTBot, however, collects content to train the company’s models, while OAI-AdsBot only visits pages submitted as ads.
In the OpenAI documentation, each ChatGPT user agent and the other OpenAI bots look like this. However, the version of the first two may change.
OAI-SearchBot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbotWhen it fetches robots.txt, OpenAI may add a robots.txt marker to the OAI-SearchBot and GPTBot user agents. Since the string changes, filter by the bot name, not the full string. In fact, the Google documentation asks the same for the Googlebot user agent: use a wildcard instead of the Chrome version.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User in logs?
OAI-SearchBot shows that ChatGPT search is crawling the site. ChatGPT-User, on the other hand, shows that a user asked a question and ChatGPT opened the page to answer. GPTBot, in turn, collects content for training. But if both GPTBot and OAI-SearchBot are allowed, OpenAI may use one crawl for both purposes.
So look at the two together before you decide that search lost interest in the site. The pages that the ChatGPT-User user agent opens, by contrast, are the ones ChatGPT fetched to answer someone. What to allow and what to block is in the guide to robots.txt for AI.
How do the AI crawlers from Claude, Perplexity, Google and Bing show up?
The other companies follow the same logic: one bot for search, another for user requests and, at some, one for training. Anthropic publishes only the bot names, while Perplexity, Google and Microsoft also publish the full user agent. In the log, look for these names:
| Bot in the log | What the visit means | Official IP list |
|---|---|---|
| Claude-SearchBot | Indexing for Claude search | bots.json, from Anthropic |
| Claude-User | A question from a Claude user | bots.json, from Anthropic |
| ClaudeBot | Collection for model training | bots.json, from Anthropic |
| PerplexityBot | Indexing for Perplexity search | perplexitybot.json |
| Perplexity-User | A question from a Perplexity user | perplexity-user.json |
| Googlebot | Google Search, including AI in search | common-crawlers.json |
| bingbot | Bing search and Copilot | bingbot.json |
At Google, the Googlebot user agent also serves AI in search. According to Google, AI is part of Search, and Googlebot rules control access. So the log does not separate the crawling for AI Overviews and AI Mode from regular search. Likewise, the Bing guidelines say that search and Copilot use the same crawling foundation.
Where do you find AI crawlers in hosting, CDN and WordPress logs?
Access logs belong to the web server, not to WordPress. On shared hosting, you usually get them from the control panel, while on your own server they sit in the file your configuration sets. Behind a CDN, however, the server records the CDN IP, not the IP of the AI crawlers, until someone changes the setup.
- Hosting with cPanel: in the Metrics section, the Raw Access tool downloads each domain’s access log as a .gz file.
- Your own server: in Apache, the CustomLog directive sets the file, and in Nginx, access_log does. Both accept the combined format, with IP, date, request, status, size, referrer and user agent.
- Managed WordPress: if the hosting panel only shows the error log, ask support for the access log.
- Cloudflare: your server logs record a Cloudflare IP, and the original IP arrives in the CF-Connecting-IP header, which Cloudflare shows you how to log. Inside Cloudflare, AI Crawl Control shows AI crawlers on all plans, as the guide on Cloudflare blocking AI crawlers explains. The Business and Enterprise plans also show live requests in Instant Logs.
How do you filter AI crawlers in logs by user agent?
Filter the log by each bot name, and then count requests, pages, days and response codes. The commands below run in the Linux and macOS terminal and, on Windows, in Git Bash. They read the combined format, which is common in Apache and the default in Nginx.
# Apache and Nginx combined format: IP in field 1, date in field 4, page in field 7 and status in field 9
LOG=access.log # compressed log (.gz): replace grep with zgrep
# 1. How many requests each bot made
for BOT in Googlebot bingbot OAI-SearchBot ChatGPT-User GPTBot ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User; do
echo "$BOT: $(grep -ci "$BOT" "$LOG")"
done
# 2. Pages that ChatGPT-User opened most
grep "ChatGPT-User" "$LOG" | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# 3. OAI-SearchBot requests per day
grep "OAI-SearchBot" "$LOG" | awk '{print substr($4, 2, 11)}' | uniq -c
# 4. Response codes that GPTBot received
grep "GPTBot" "$LOG" | awk '{print $9}' | sort | uniq -c | sort -rn
# 5. IPs that used the Googlebot name, to check later
grep "Googlebot" "$LOG" | awk '{print $1}' | sort | uniq -c | sort -rn | headThe first command ignores case, because Bing writes bingbot in lowercase in its user agent. Command 2 shows the pages that the ChatGPT-User user agent opened, that is, what ChatGPT fetched to answer users. Command 5, in turn, lists the IPs that used the Googlebot user agent. To see what AI crawlers request beyond the HTML, such as scripts and API calls, use the JavaScript SEO checklist.
What do 403 and 429 codes mean for AI crawlers in logs?
The response code shows what each bot received. In general, 200 means the page was delivered, 301 and 302 mean a redirect, and 404 means the page does not exist. However, a 403 or a 429 for only one bot points to a block outside robots.txt, such as a firewall, the CDN or a host rate limit.
At NotHuman, a test of ours left a question that the log helps answer. Our host returned 429 to a request from a regular IP with the full GPTBot user agent, even for robots.txt. Requests with the OAI-SearchBot, ChatGPT-User, ClaudeBot and browser user agents got 200. But the test does not reproduce the IP, so we don’t know if the real bot, coming from OpenAI IPs, also gets 429.
So test each user agent and check the logs, because the block may come from the host, not from robots.txt or WordPress. In your server logs, the answer lies in the pair of user agent and IP. A 429 on GPTBot lines from IPs on the OpenAI list shows that the real bot also gets blocked.
How do you confirm AI crawlers are real with the published IPs?
Compare the IP on each log line with the lists that the companies publish. After all, anyone can write a bot name in a request, including the Googlebot user agent. For Google and Bing, there is also reverse DNS: the IP’s hostname ends in the company’s domain and points back to the same IP.
Anthropic and Perplexity, for example, point to this method. The Python 3 script below downloads the official lists and tells you which bot each IP in the log belongs to.
# check_ip.py: tells which official list each IP belongs to
# Usage: python check_ip.py 66.249.66.1 203.0.113.5
import ipaddress, json, sys, urllib.request
LISTS = {
"OAI-SearchBot": "https://openai.com/searchbot.json",
"ChatGPT-User": "https://openai.com/chatgpt-user.json",
"GPTBot": "https://openai.com/gptbot.json",
"Anthropic (Claude)": "https://claude.com/crawling/bots.json",
"PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
"Perplexity-User": "https://www.perplexity.com/perplexity-user.json",
"Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
"Bingbot": "https://www.bing.com/toolbox/bingbot.json",
}
networks = {}
for bot, url in LISTS.items():
request = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
data = json.load(urllib.request.urlopen(request, timeout=30))
networks[bot] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"), strict=False)
for p in data["prefixes"]]
for ip in sys.argv[1:]:
address = ipaddress.ip_address(ip)
owners = [bot for bot, nets in networks.items() if any(address in net for net in nets)]
print(ip, "->", ", ".join(owners) if owners else "not in the published lists")Run the script with the IPs from command 5. An IP that uses a bot name but sits outside that bot’s list points to a fake. For Google and Bing, however, confirm with reverse DNS before you decide. The lists also change, and Bing, for example, asks you to refresh its list every day.
How do you verify Googlebot and Bingbot with reverse DNS?
Run two DNS lookups. First, look up the IP from the log: the name must end in googlebot.com, google.com or googleusercontent.com, according to Google, or in search.msn.com, according to Bing. Then look up that name and check that it points back to the same IP.
nslookup 66.249.66.1
# the name must end in googlebot.com: crawl-66-249-66-1.googlebot.com
nslookup crawl-66-249-66-1.googlebot.com
# the address must be the same IP as in the log: 66.249.66.1In the examples, Google uses the host command, and Bing also shows nslookup on Windows. So, before you trust a Googlebot or Bingbot user agent, run both lookups. Bing also offers the Verify Bingbot tool. For OpenAI, Anthropic and Perplexity, however, the official pages only point to the IP lists.
Which tool should you use to analyze SEO and AI crawler logs?
To start, the terminal and a spreadsheet are enough, because the commands above sum up the log by bot, page and day. For regular analysis, an SEO log tool brings together counts, codes and bot checks. Search Console, in contrast, shows only Google’s crawlers, and the CDN shows the AI crawlers that pass through it.
- Screaming Frog Log File Analyser: reads the log, verifies search and AI bots and shows crawled URLs, frequency and response codes. The free version analyzes up to 1,000 lines, and the yearly license removes the limit.
- Search Console crawl stats: under Settings, the report shows how many requests Google made, when, and with what response. This data comes from Google, not from the Googlebot user agent in your log, so fakes stay out.
What don’t logs show about AI crawlers?
Logs show the visit, but not what the AI did with the page. They don’t say if the text went into an answer, became a citation, or which question led the bot there. In addition, some requests never reach the server, so they stay out of the logs.
- Citations: in Bing Webmaster Tools, AI Performance shows how often each page became a source in Copilot and in Bing’s AI summaries.
- Gemini training: Google-Extended has no user agent of its own, because Google uses the token only as a control in robots.txt. So it does not show up in the log.
- Cache and CDN: Cloudflare caches robots.txt by default, and a bot blocked at the CDN shows up only in the CDN’s records. In both cases, the request is missing from your server logs.
- Clicks from people: someone who clicks a link in ChatGPT arrives with their own browser’s user agent. To measure those clicks, see the guide to AI traffic in GA4.
What if AI crawlers never show up in your logs?
First, check that the log is complete: domain, period and real IP behind the CDN. Then look for the block in three layers: robots.txt, the CDN and the host. Finally, remember that ChatGPT-User and Perplexity-User only show up when someone asks a question that leads the assistant to the site.
- Gather the files. Download the logs for each domain, with and without www, and the .gz files from past months.
- Read robots.txt. Look for Disallow under OAI-SearchBot, Claude-SearchBot and PerplexityBot or in the group for all bots.
- Open the CDN and the firewall. A bot blocked there never reaches your server logs.
- Test each user agent. Compare each bot’s response with a browser’s, as in the NotHuman test above.
- Go back to the log after the fix. OpenAI says it takes about 24 hours for ChatGPT search to reflect a robots.txt change.
This diagnosis is one item of the technical SEO audit, which also covers indexing. After all, crawler access is the technical base of SEO for the AI era.
Frequently asked questions
Not as visits, because Google Analytics automatically excludes traffic from known bots and spiders, and you cannot turn this off. So AI crawler visits only appear in your server and CDN logs. Clicks from people coming from ChatGPT, however, do show up in GA4.
No. In the combined format, each line has IP, date, request, status, size, referrer and user agent, but no field stores the question. The log only shows the page that ChatGPT-User opened to answer. So the pages it opens most point to the topic, not the exact question.
It depends on the host. In cPanel, the Raw Access tool can archive the logs in the account’s logs folder and, by default, keeps those files with no time limit. However, there are options to delete the previous month’s logs or to set a retention period. To compare months, download the files every month.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- OpenAI, Overview of OpenAI Crawlers (2026).
- Anthropic, Anthropic’s crawlers and how to control them (2026).
- Perplexity, Perplexity Crawlers (2026).
- Google Search Central, verifying requests from Google’s crawlers and fetchers (2026).
- Google Search Central, Google’s common crawlers (2026).
- Bing Webmaster Tools, How to Verify Bingbot (2026).
- cPanel, Raw Access (2026).
- Cloudflare, Restoring original visitor IPs (2026).
