Reading time: 12 minutes
The llms.txt file introduces a website to language models and AI agents. It’s a Markdown file with a summary and links to the main pages. However, no major AI engine has said it uses the file to choose what to cite. See the data and how Yoast or Rank Math create llms.txt in WordPress.
In short
- Google says Search ignores llms.txt. So creating the file neither helps nor hurts, including in Search’s AI features.
- According to an Ahrefs study of 137K domains, 97% of llms.txt files got no requests at all in May 2026.
- Of the requests that did arrive, AI search bots made 1.1%. On the other hand, AI agents such as Claude Code, plus the infrastructure behind them, made 10.5%.
- In WordPress, Yoast and Rank Math generate llms.txt for free. It’s worth turning on and reviewing, but not paying for as a GEO tactic.
What is llms.txt and what is it for?
llms.txt is a Markdown text file published at yoursite.com/llms.txt. It summarizes the site and lists links to its most useful pages. So it guides AI agents and models when they need information about the site. However, it doesn’t block or allow anything, and it doesn’t replace the sitemap.
Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024. According to the proposal’s site, version 2 dates from August 2026. In it, the file can sit at the root or in a subfolder, such as /docs/, and it covers the pages under that path. Only one part is required: an H1 title with the site’s name. The rest is optional:
- a short summary in a blockquote, on the line that starts with “>”;
- context paragraphs or lists, without headings;
- H2 sections with lists of links, each link with an optional note;
- an “Optional” section, by convention, for links the agent can skip.
For example, a fictional file in the proposal’s format would look like this:
# Example Dental Clinic
> Dental clinic in Campinas, Brazil, with private and insurance-based care.
## Services
- [Dental implants](page URL): types, timelines and price range
- [Accepted insurance plans](page URL): updated list and coverage rules
## Optional
- [Blog](blog URL)But llms.txt isn’t robots.txt for AI. According to the proposal, robots.txt tells bots what access they may have. An agent, on the other hand, reads the file on demand, that is, when it needs information to help someone. Likewise, it isn’t a sitemap: the sitemap lists every indexable page, while llms.txt offers a short selection. To control access, see which AI crawlers to allow and which to block.
The llms-full.txt file, in turn, isn’t part of the proposal. Even so, documentation platforms such as Mintlify generate this second file with the entire documentation in one place.
Does any AI engine use llms.txt?
No major AI engine has said it uses sites’ llms.txt files to search or cite content. Google says Search ignores the file, including in its AI features. Also, the crawler pages of OpenAI, Anthropic and Perplexity handle access through robots.txt and don’t mention the file. Today, the most common use of the format is in technical documentation.
Google’s guide to generative AI in Search, published in May and updated in July 2026, is blunt. According to the guide, you don’t need to create AI text files, because Search doesn’t use them. Therefore, creating an llms.txt will “neither harm nor help” your visibility in Search. Google may crawl the file like any other, but without special treatment.
In a Reddit discussion reported by Search Engine Journal in June 2026, Google’s John Mueller called the topic speculative. After all, he said, the file has existed for years, yet no AI system uses it. What Google actually asks for is in the guide on how to show up in AI Mode and AI Overviews.
What do OpenAI, Anthropic, Perplexity and Chrome say?
On OpenAI’s crawler page, you control OAI-SearchBot and GPTBot access through robots.txt. The pages from Anthropic and Perplexity follow the same model, and none of the three says their bots read sites’ llms.txt files. In fact, all three companies have their own llms.txt, but only in their developer documentation.
Finally, there’s a signal from Chrome. Lighthouse has an experimental agentic browsing category with an llms.txt audit. According to the documentation, without the file, agents may spend more time understanding the site. Even so, the audit only flags server errors. If the site has no file, Lighthouse marks the audit as not applicable, since the file is optional for now.
What does the request data show?
The data shows that almost no one requests the file. For example, in the Ahrefs study of 137K domains, 97% of published files got no requests in a month. Also, among the requests that did arrive, AI search bots barely show up. The three studies below come from companies that sell SEO or AI monitoring tools.
| Study | Sample | Result |
|---|---|---|
| Ahrefs, June 2026 | 137K domains, May 2026 | 97% of files with no requests |
| OtterlyAI, February 2026 | 1 site, 90 days | 84 of 62.1K AI bot visits |
| SE Ranking, November 2025 | About 300K domains | No link to AI citations |
What did Ahrefs find?
Ahrefs analyzed traffic to 137,210 domains in May 2026. Of those, 28% published the file, but the company treats that number as a ceiling, because Ahrefs Web Analytics customers are more technical than average. Of the published files, 97% got no hits that month.
The other 3%, about 1.1K domains, received 22K requests, and bots made 96% of them. However, AI search bots such as OAI-SearchBot and PerplexityBot made only 1.1%. Slackbot alone requested the file more often than PerplexityBot. Meanwhile, Claude Code, Anthropic’s coding agent, requested it more than any AI search bot or assistant.
On sites without the file, requests that returned a 404 error came almost entirely from people, and none from AI bots. In other words, no AI bot looked for a file that didn’t exist. Finally, Ahrefs notes that requesting the file doesn’t prove a bot used its content.
What do OtterlyAI and SE Ranking show?
OtterlyAI published the file on a test site and tracked its logs for 90 days. During that period, the file got 84 AI bot visits, about 0.1% of the total. By comparison, each content page on the site got about 265 on average.
SE Ranking, in turn, compared the presence of the file with how often AI answers cited each domain, across about 300K domains, and found no relationship. In fact, removing that variable from its prediction model even improved accuracy. Likewise, the guide to what GEO is lists llms.txt among tactics with no evidence of earning citations.
How do Yoast and Rank Math generate llms.txt in WordPress?
In WordPress, Yoast and Rank Math generate llms.txt for free from your published content. Yoast turns it on with one button and picks the pages itself, but it allows manual selection. Rank Math, by contrast, asks you to choose the content types, because the file stays empty until you configure it. In both cases, the file shows up at yoursite.com/llms.txt.
| Item | Yoast SEO | Rank Math |
|---|---|---|
| How to turn it on | Button in site features | LLMS Txt module in the dashboard |
| What goes in | 5 recent pieces of content and 5 categories or tags | Chosen content types and taxonomies |
| Manual tweaks | Page selection | Extra links and text |
| What’s left out | Posts older than 12 months | Posts set to noindex |
How does llms.txt work in Yoast?
Yoast SEO, a WordPress SEO plugin, has generated llms.txt since version 25.3, from June 2025. The feature sits in the site features settings, under AI tools. According to the Yoast specification, the plugin creates llms.txt at the root and updates it every week.
The Yoast llms.txt includes the 5 most recently updated posts, pages or custom post types, with priority for cornerstone content. However, it only includes posts published in the last 12 months. It also includes the 5 categories or tags with the most content. Finally, each post’s description comes from its excerpt, when there is one.
If there’s already an llms.txt on the server, Yoast doesn’t overwrite it and shows a warning. So, to use the plugin’s file, delete the old one. Also, the Yoast llms.txt doesn’t work on multisite installations.
At NotHuman, our blog runs on WordPress, and Yoast publishes the automatic llms.txt. The same plugin notifies IndexNow and generates the BlogPosting and BreadcrumbList schema for our posts. Also, the text of our posts already comes in the initial HTML, and our robots.txt doesn’t block GPTBot, Google-Extended or CCBot.
How does llms.txt work in Rank Math?
Rank Math, another WordPress SEO plugin, generates llms.txt through its LLMS Txt module, which you turn on in the plugin dashboard. In it, you choose content types and taxonomies, set an item limit and can add links and text by hand. The default limit is 100 items. Also, the file leaves out posts set to noindex. In September 2026, the free version updated the file to match the latest specification. For the other AI features of Yoast and Rank Math, see what GEO plugins for WordPress do.
To check the llms.txt on your WordPress site, open yoursite.com/llms.txt in your browser and see whether the listed pages are the ones you want to show. Then search for /llms.txt in your server or CDN logs to see who requests the file.
When is llms.txt worth creating?
It’s worth creating when the cost is close to zero or when there’s a likely reader. For example, that’s the case for sites with technical documentation or an API that coding agents consult. Likewise, it’s worth it in WordPress, where the plugin generates llms.txt for free. But it’s not worth paying for the file in the hope of showing up in ChatGPT.
According to the proposal, software documentation is the most common use. There, coding agents follow the file’s links to find API references and tutorials. The Ahrefs data points the same way, since Claude Code is among the top readers. Ahrefs also says agents request the file when something points them to it. So it’s worth linking the file from your documentation pages.
For other sites, Google’s John Mueller offered a practical rule of thumb:
“When an AI platform that brings you clients complains that it needs the file for your site, then I’d recommend taking the time to create one.”
John Mueller, Google (2026)
If your plugin already generates the file, review what it publishes. After all, Ahrefs found a research bot that studies llms.txt as a gateway for prompt injection. In other words, the risk is hidden instructions in the text that manipulate agents. So the company recommends treating the file like code. Keep only links and descriptions, nothing that looks like an instruction, and control who can edit it.
What if your brand doesn’t show up in AI answers?
If your brand doesn’t appear in AI answers, a missing llms.txt doesn’t explain the problem. In general, the cause lies elsewhere, as the 8 reasons AI doesn’t cite a brand show. In fact, in the same discussion, Mueller said the biggest hurdle for most sites is making sure they don’t block agents.
So start with access. First, check whether your CDN lets search bots through, as in the guide on how to check and allow AI crawlers in Cloudflare. Then check whether your text arrives in the initial HTML, as the guide on JavaScript SEO for AI crawlers explains.
What we found when we analyzed 15 Brazilian pages about llms.txt
We read 15 pages in Brazilian Portuguese that showed up in 3 searches about llms.txt. The searches covered what the file is, whether it works and how to create llms.txt in WordPress. Nine pages say Google doesn’t use the file. However, only three give any request numbers, and only one cites the Ahrefs study.
Also, four claim that the file increases, or may increase, visibility or citations in AI. Two say it controls what AI can use, although the proposal doesn’t provide for that. On the other hand, none mentions the risk of prompt injection. Six have no update after Google’s May 2026 guide. Finally, two still say Rank Math didn’t have the feature.
Frequently asked questions
No. The robots.txt file says what bots may crawl, while llms.txt doesn’t block or allow anything. According to the proposal, an agent reads the file on demand, when it needs information about the site. So to control AI bots, the rules still live in robots.txt and in your CDN.
The llms.txt file is a short index, with a summary of the site and links to its main pages. The llms-full.txt file, by contrast, isn’t part of the proposal. Even so, documentation platforms such as Mintlify generate that file with the entire documentation in one place.
Not as a tactic to show up in ChatGPT or Google, because no major AI engine has said it uses the file. Also, in the Ahrefs study, 97% of llms.txt files got no requests in May 2026. Finally, in WordPress, Yoast and Rank Math generate llms.txt for free.
Not in Google Search. Google says creating the file neither harms nor helps visibility, because Search ignores it. However, there’s another concern: review the content of your llms.txt. After all, some people already study the file as a gateway for prompt injection into AI agents.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- llmstxt.org, The /llms.txt file, proposal by Jeremy Howard, version 2 (2026).
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (2026).
- Google Search Central, Optimizing your website for generative AI features on Google Search (2026).
- Search Engine Journal, Google Says LLMs.txt Is Purely Speculative… For Now (2026).
- Yoast, Yoast SEO llms.txt functional specification (2026).
- Rank Math, How to Use llms.txt in Rank Math SEO (2025).
- OtterlyAI, 90-day llms.txt experiment (2026).
- SE Ranking, analysis of llms.txt and AI citations across about 300K domains (2025).
