Robots.txt checker: how to test robots.txt in Google, Bing and AI
How to test robots.txt now that the old Search Console tester is gone: the robots.txt report, URL Inspection, the Bing tester…
Enter your domain and use the robots.txt tester to see, in seconds, what Google, ChatGPT, Claude and Perplexity can read.
Free, no signup
It is a text file at the root of a site that tells crawlers which paths they may access. Google follows these rules when crawling, and AI crawlers such as GPTBot and ClaudeBot have their own.
But it is not a way to keep a page out of Google: according to Google’s documentation, that takes noindex or password protection.
User-agent: *
Disallow:User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Want more templates? See the robots.txt template for AI.
How to test robots.txt now that the old Search Console tester is gone: the robots.txt report, URL Inspection, the Bing tester…
Which AI crawlers to allow and which to block in robots.txt, based on documentation from OpenAI, Anthropic, Perplexity, Google…
Three ready robots.txt blocks to copy, with what each one does: allow the AI bots, block only training and block the listed AI…
The list of AI bot user agents that the companies document themselves: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and more…
The diagnosis for when you can’t find your site on Google: how to confirm indexing, how long to wait, the most common technical…
The 8 most common reasons AI doesn’t cite a brand, from a crawler blocked in Cloudflare to data that differs between site and…
In 2023 Google replaced the old Search Console tester with a robots.txt report that lists the files found, errors and warnings, but does not test URLs. Therefore, this tool fills that gap with the rules Google documents.
No. It is an independent implementation based on Google’s public documentation. To see how Googlebot treats a specific URL, use the URL Inspection tool in Search Console.
Not necessarily. According to OpenAI, GPTBot collects content that may train models and OAI-SearchBot decides whether the site shows up in ChatGPT search; the two settings are independent. So, to stay out of search, the block goes on OAI-SearchBot.
Anthropic separates ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests). Google-Extended controls the use of content in Gemini and does not affect Google Search.
If the server answers 404 or another 4xx (except 429), Google treats it as no restrictions and crawls the whole site.
The most specific rule wins, meaning the longest path. If both are the same length, the less restrictive one wins: Allow.
Yes. Each host has its own robots.txt: blog.example.com is separate from example.com. So, check them one at a time.
No. robots.txt controls crawling, not indexing or citation. Besides, a firewall or CDN can also block crawlers without leaving a trace in the file. So, measuring whether AI cites your brand, against your competitors, is NotHuman’s job.
Book a demo and see how your brand shows up in AI answers, with support from a GEO specialist.
Book a demoWe use necessary cookies to make the site work and Google Analytics and Microsoft Clarity statistics cookies to improve our content. You can decline the statistics ones; in the EU, UK and Switzerland they are only turned on after you accept. Cookie Policy.