Robots.txt tester: can Google and AI read your site?

Enter your domain and use the robots.txt tester to see, in seconds, what Google, ChatGPT, Claude and Perplexity can read.

Free, no signup

How to use the robots.txt tester

  1. First, enter the domain and click Analyze. The tool fetches the site’s file, which always lives at /robots.txt.
  2. Then, read the verdict for Google, AI search and AI training.
  3. Finally, open “View the robots.txt and test URLs” to check a specific URL or try a rule before you publish it.
Robots.txt tester: result of an example, with Google open and OAI-SearchBot blocked among the AI crawlers

What is robots.txt?

It is a text file at the root of a site that tells crawlers which paths they may access. Google follows these rules when crawling, and AI crawlers such as GPTBot and ClaudeBot have their own.

But it is not a way to keep a page out of Google: according to Google’s documentation, that takes noindex or password protection.

robots.txt examples

Allow everything

User-agent: *
Disallow:

Block GPTBot and keep OAI-SearchBot

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Want more templates? See the robots.txt template for AI.

Blog

Articles about robots.txt and AI

Common questions

Frequently asked questions about the robots.txt tester

Does Google still have a robots.txt tester?

In 2023 Google replaced the old Search Console tester with a robots.txt report that lists the files found, errors and warnings, but does not test URLs. Therefore, this tool fills that gap with the rules Google documents.

Is this an official Google tool?

No. It is an independent implementation based on Google’s public documentation. To see how Googlebot treats a specific URL, use the URL Inspection tool in Search Console.

Does blocking GPTBot remove my site from ChatGPT?

Not necessarily. According to OpenAI, GPTBot collects content that may train models and OAI-SearchBot decides whether the site shows up in ChatGPT search; the two settings are independent. So, to stay out of search, the block goes on OAI-SearchBot.

What does each AI crawler do?

Anthropic separates ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests). Google-Extended controls the use of content in Gemini and does not affect Google Search.

What happens if my site has no robots.txt?

If the server answers 404 or another 4xx (except 429), Google treats it as no restrictions and crawls the whole site.

Allow or Disallow: which one wins?

The most specific rule wins, meaning the longest path. If both are the same length, the less restrictive one wins: Allow.

Do I need to test each subdomain?

Yes. Each host has its own robots.txt: blog.example.com is separate from example.com. So, check them one at a time.

Does an open robots.txt guarantee AI will cite my brand?

No. robots.txt controls crawling, not indexing or citation. Besides, a firewall or CDN can also block crawlers without leaving a trace in the file. So, measuring whether AI cites your brand, against your competitors, is NotHuman’s job.

Demo

Opening access is only the first step

Book a demo and see how your brand shows up in AI answers, with support from a GEO specialist.

Book a demo

Your privacy

We use necessary cookies to make the site work and Google Analytics and Microsoft Clarity statistics cookies to improve our content. You can decline the statistics ones; in the EU, UK and Switzerland they are only turned on after you accept. Cookie Policy.

Cookie preferences

Cookie Policy