Reading time: 12 minutes
A robots.txt checker shows whether a robots.txt rule stops a crawler from crawling a URL. To test robots.txt in Google and Bing, use Search Console, with URL Inspection, and the Bing Webmaster Tools tester. For AI crawlers, however, test the rule and also the server response.
In short
- In November 2023, Google replaced the Search Console robots.txt tester with a report. So, to test a URL, it points to URL Inspection.
- Bing Webmaster Tools, on the other hand, still has a tester: it shows the line that blocks the URL and tests changes before you publish.
- OpenAI, Anthropic and Perplexity offer no tester. So, for GPTBot and the others, use a third-party robots.txt checker and curl.
- robots.txt is only one layer. After all, the CDN, the firewall and the host can block a crawler that the file allows.
How do you test robots.txt now that the old Search Console tester is gone?
In Google, testing now has two parts. The Search Console robots.txt report shows the file Google read, when it read it and the errors. URL Inspection, in turn, tells you whether a rule blocks a page. In fact, this report took the place of the old Search Console robots.txt tester in November 2023.
According to the Search Console help, the report lists the files for the top 20 hosts of the site. But it only works in Domain properties or in URL-prefix properties without a path. Errors prevent a rule from being used, while warnings don’t.
After you fix an error, request a recrawl in the report itself, although that doesn’t guarantee an immediate visit to the unblocked URLs. To test robots.txt before you publish, Google points to its open source library, which Google Search uses and which is aimed at developers.
How do you know if a URL is blocked for Googlebot?
Use URL Inspection, because the “Crawl allowed?” field shows whether a robots.txt rule blocks the page. After you change robots.txt, click “Test live URL”, because the default result comes from the last indexed version.
In addition, the live test helps you find blocked CSS and JavaScript. According to the URL Inspection help, differences in the screenshot might come from blocked resources. But each property has a daily limit for these tests.
How do you see if a rule blocks Googlebot, Bingbot or GPTBot?
To test robots.txt crawler by crawler, you need a different tool for each one. For Googlebot, use URL Inspection, and for Bingbot, use the Bing Webmaster Tools tester. For GPTBot and the other AI crawlers, however, there is no official tool: use a third-party robots.txt checker and confirm with curl.
| Crawler | Where to test | What the test shows |
|---|---|---|
| Googlebot | Search Console URL Inspection | Whether robots.txt blocks the URL |
| Bingbot and AdIdxbot | Bing Webmaster Tools tester | The line that blocks the URL |
| GPTBot, ClaudeBot and others | Third-party checker | The rule that applies, as the tool reads it |
| Any crawler | curl with the user agent | Whether the server answers the crawler |
First, find out which group applies to each crawler. Under Google’s specification and RFC 9309, the robots.txt standard, a crawler follows only the group with its name. If there is none, it follows the * group. Which AI crawlers to allow is covered in the guide to robots.txt for AI.
Bingbot: the Bing Webmaster Tools robots.txt tester
Bing Webmaster Tools still has a robots.txt tester. According to the Bing help, you enter the URL and choose between Bingbot and AdIdxbot. Then you see whether a rule blocks the URL, which line does it and for which crawler. However, the test uses the text in the editor, which can include changes you haven’t published yet.
So, in Bing, you can test the edited robots.txt before you publish it. Then download the file, upload it to the site root and ask Bing to read the new version. If the site isn’t in Bing Webmaster Tools yet, see how to index your site on Bing.
GPTBot and other AI crawlers: third-party checkers
OpenAI, Anthropic and Perplexity explain the rules for their crawlers, but they offer no tester. So, to test robots.txt with GPTBot, ClaudeBot or PerplexityBot, use a third-party robots.txt checker that accepts any user agent. In fact, the Search Console help also tells you to use a robots.txt validator, without saying which one.
Treat the result as a first pass, because, according to Google, each implementation parses robots.txt slightly differently. Google-Extended, for its part, has no user agent of its own, so you test it by reading the file.
How do you test the server with curl and each AI crawler’s user agent?
Request robots.txt and an important page with each crawler’s user agent, then compare the results with a browser. This test, however, doesn’t read the rules in the file. Instead, it shows whether the server, the CDN or the firewall answers each crawler differently, which no robots.txt checker can see.
The user agents are the ones published by OpenAI, Perplexity, Google and Bing. Anthropic, on the other hand, publishes only the crawler names, which work for the test. Run it in the Mac or Linux terminal, or in Git Bash on Windows, with your own SITE and PAGE.
SITE="https://www.yoursite.com"
PAGE="/" # replace with an important page, such as /category/product
while IFS='|' read -r BOT UA; do
R=$(curl -sL -m 30 -o /dev/null -A "$UA" -w "%{http_code} %{content_type}" "$SITE/robots.txt")
P=$(curl -sL -m 30 -o /dev/null -A "$UA" -w "%{http_code}" "$SITE$PAGE")
echo "$BOT | robots.txt: $R | page: $P"
done <<'LIST'
Browser|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36
Googlebot|Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Bingbot|Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Claude-SearchBot|Claude-SearchBot
ClaudeBot|ClaudeBot
LISTHow do you read the test result?
Compare each crawler with the browser line. If only the crawler gets an error, the block is in the server, the CDN or the firewall, and not in robots.txt. An error in the file itself, however, changes how the crawler treats the whole site.
| Result | What it means | What to do |
|---|---|---|
| 200 and text/plain for robots.txt | The crawler gets the rules | Test the robots.txt rules per crawler |
| 404 for robots.txt | No file, everything allowed | Nothing, if you don’t want to block |
| 200 and text/html for robots.txt | Probably a page instead of the file | Publish a plain text file |
| 5xx or 429 for robots.txt | Server error for Google | Fix the server first |
| 403, 429 or 000 (no response) for one crawler only | Block outside robots.txt | Check the CDN, firewall and host |
According to Google’s specification, if robots.txt returns a 5xx, Google stops crawling the site for the first 12 hours. For Google, a 429 also counts as a server error. Other 4xx errors on the file, in contrast, work as if there were no robots.txt. Under RFC 9309, a 5xx error also requires crawlers to assume a complete disallow.
The test, however, comes from your own IP. Google and Bing warn that anyone can fake a user agent, and the server can check the IP. So a 403 for the test’s Googlebot may just be that check. Likewise, a firewall that allows PerplexityBot by user agent and IP, as Perplexity recommends, can block the test and let the real crawler in. That is why only the logs show what the real crawler received.
What doesn’t a robots.txt checker catch?
No robots.txt checker sees what happens before the file. CDN, firewall or hosting rules can block a crawler that robots.txt allows. In addition, the Google and Bing testers don’t simulate AI crawlers. So curl and the logs complete the diagnosis.
At NotHuman, a user agent test on our own site shows why you should test the server. In a request sent from a regular IP with GPTBot’s full user agent, the hosting server returned a 429, even for robots.txt. On the other hand, the requests with the user agents of OAI-SearchBot, ChatGPT-User, ClaudeBot and a browser got a 200. But the test doesn’t reproduce the IP, so we don’t know whether the real crawler, coming from OpenAI’s IPs, also gets a 429. Therefore, testing each user agent and checking the logs is part of testing robots.txt: the block can sit with the host, outside the file and WordPress.
On sites behind Cloudflare, the AI policies and the firewall can block search crawlers even when robots.txt allows them. See how to check and allow AI crawlers on Cloudflare. Blocking IPs also has a side effect: according to Anthropic, blocking its crawlers’ IPs can stop them from reading robots.txt. After all, crawler access is the technical base of SEO for the AI era. That is why the robots.txt test opens a technical SEO audit.
Which common robots.txt mistakes block the whole site?
Three mistakes can close the whole site: Disallow: / in the * group, a wildcard that turns into a slash and a server error, which the curl test shows. Others hide only part of the site, such as new groups that override the general rules and blocked CSS or JavaScript.
Disallow: /in the*group. The slash alone blocks the site for every crawler without its own group, for example when the staging file goes live. An emptyDisallow:, however, blocks nothing, because Google ignores rules without a path.- A wildcard that turns into a slash. Under Google’s specification,
Disallow: /*is the same asDisallow: /. By the same wildcard rule,Disallow: /*?blocks every URL with a parameter, including filters you want in Google. - A new group that overrides the general rules. A group for Googlebot or GPTBot makes that crawler ignore the
*group. After you create the group, test robots.txt again with the URLs that the*group blocked. - Blocked CSS and JavaScript. Google asks you not to block resources that the page analysis needs. Check them in URL Inspection and, for AI crawlers, in the JavaScript SEO checklist.
Robots.txt or noindex: which one removes pages from search?
To remove a page from search, use noindex and let the crawler crawl the URL. The robots.txt file controls crawling, not indexing: a blocked URL can still appear in Google, without a description, if other pages link to it. To reduce the load on your server, however, robots.txt is the right tool.
According to the noindex documentation, the crawler only sees the tag if it can crawl the page. So, with the URL blocked, URL Inspection shows “Yes” for “Indexing allowed?”. In the Page indexing report, the page may show up as “Indexed, though blocked by robots.txt”. In that case, Google tells you to remove the block and use noindex. Also, Google doesn’t support noindex inside robots.txt.
In AI, a robots.txt block doesn’t erase the URL either. According to the Bing guidelines, robots.txt controls access, not indexing, and noindex removes a URL from Bing and Copilot. At OpenAI, in turn, a site that blocks OAI-SearchBot drops out of ChatGPT search answers, but it can still appear as a navigational link.
How do Wix, VTEX and WordPress create robots.txt?
On all three platforms, you change robots.txt in the dashboard, with an app or with a plugin. So the way to test robots.txt is the same on all of them. Open the live /robots.txt and run the key URLs through Search Console and the Bing tester. What changes, however, is where you edit it and what comes by default.
Wix. Wix creates the file and, by default, allows all pages, except password-protected pages, members-only pages and pages set to noindex. According to the Wix help, the file updates whenever you publish a page. To edit it, use the Robots.txt Editor under SEO & GEO. When you see the “Blocked by robots.txt” error, Wix tells you to check those settings instead of editing the file.
VTEX. According to the VTEX help, the platform has a native interface, under Store Settings > Storefront > Settings > SEO > Robots.txt. VTEX asks you to check whether each rule is needed, whether it catches the right URLs and whether the file lists the sitemap. In a file full of Disallow lines, test one URL of each type: category, product and filter.
WordPress. WordPress creates a default robots.txt that blocks /wp-admin/ and allows admin-ajax.php, and plugins can change that text. Since version 5.3, however, the option that asks search engines not to index the site no longer writes Disallow: /, according to the WordPress documentation. Instead, it adds noindex to the pages: a robots.txt checker doesn’t see it, but URL Inspection does.
Frequently asked questions
No, because OpenAI and Anthropic publish their crawler names and explain how to block them, but they offer no tester. For these crawlers, use a third-party checker that accepts any user agent, or Google’s open source library. Then confirm with curl that the server returns a 200 to the crawler.
Usually up to 24 hours, because Google generally caches robots.txt for that long. After you fix an error, request a recrawl in the Search Console robots.txt report. Even so, the request doesn’t guarantee an immediate crawl of the unblocked URLs.
No, because Google-Extended has no user agent of its own: Google crawls with its regular crawlers and uses the token only as a control in robots.txt. So you test Google-Extended by reading the file. Check that its group has the rule you want.
It covers the crawler that feeds both. According to the Bing guidelines, Bing search and Copilot rely on the same crawling and indexing foundation. So testing Bingbot in the Bing robots.txt tester also covers the crawling that serves Copilot.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- Search Console Help, robots.txt report (2026).
- Search Console Help, URL Inspection tool (2026).
- Google Search Central, how Google interprets the robots.txt specification (2026).
- Bing Webmaster Tools, robots.txt tester help (2026).
- OpenAI, Overview of OpenAI Crawlers (2026).
- Perplexity, Perplexity Crawlers (2026).
- Google Search Central, block indexing with noindex (2025).
- IETF, RFC 9309, Robots Exclusion Protocol (2022).
