Google and other major AI systems do not currently use the llms.txt file, according to a 2025 statement from John Mueller. Implementing the full proposal creates a maintenance paradox where owners must maintain a secondary plain-text version of every important page for a reader that does not exist. Technical SEO specialists should act on this by focusing on on-page content structure.
Who should act on llms.txt and who should skip it
Technical SEO specialists should skip building llms.txt mirrors until Google or OpenAI confirm support
- Technical SEO specialists should skip building llms.txt mirrors until a major provider like Google or OpenAI officially confirms support.
- Technical SEO specialists should instead verify robots.txt configurations for specific AI agents like GPTBot and ClaudeBot.
Content strategists should focus on clear headings and direct answers for AI retrieval
- Content strategists should skip the llms.txt file and focus on creating clear headings and direct answers for AI retrieval.
- Content strategists should prioritize consistent naming conventions to improve LLM visibility.
In-house SEO managers should skip llms.txt due to its lack of measurable ROI
- In-house SEO managers should skip llms.txt as it currently offers no measurable ROI for AI search results.
SEO agency staff should skip recommending llms.txt because robots.txt remains the standard
- SEO agency staff should skip recommending llms.txt as a primary tactic and instead explain that robots.txt remains the standard for bot control.
What to do next about the llms.txt file
- Verify the robots.txt file for GPTBot and ClaudeBot user agents to establish proper AI crawler instructions.
- Audit on-page heading tags and direct answers to improve retrieval efficiency for AI Overviews.
The llms.txt proposal is more than a simple index file
The llms.txt proposal, which emerged in late 2024, suggests placing a markdown file at the root of a website to provide a short site description and a curated list of important pages. Many practitioners stop at this index file, but the full proposal requires creating a markdown copy of every important page on the site with a.md extension. This requirement forces the site owner to maintain a second, plain-text version of the entire website alongside the live version.
What is llms.txt?
llms.txt is a proposed LLM markdown file located at a site’s root directory that provides a summary of the website’s content. It serves as a shortcut for large language models to understand the site’s purpose and primary pages without crawling every single URL. For most users, the file currently serves as a form of “perceptual SEO” because it provides a sense of preparation even though no one is reading it.
Why llms.txt is not a replacement for robots.txt
The llms.txt file does not function as a mechanism to block or allow crawlers, which is a common misconception among those seeking AI crawler instructions. Robots.txt remains the industry standard for controlling bot access via specific user-agent directives like GPTBot and ClaudeBot. While llms.txt is a content discovery proposal, robots.txt is a functional access control tool.
Key terms for llms.txt, robots.txt, and Sitemap.xml
- llms.txt
- A proposal for a markdown-based content index to help models understand site structure.
- robots.txt
- The standard method for blocking or allowing specific bots via user-agent directives.
Sitemap.xml also remains a relevant standard, providing a structured XML schema for discovery that differs from the markdown-based llms.txt proposal. Because llms.txt is not a standard like robots.txt, it currently provides no functionality for blocking AI bots from scraping data.
Data proves current AI systems ignore llms.txt
Server logs provide clear evidence that major AI companies are not currently utilizing the llms.txt file. One site tracked 500 million AI bot visits over three months but received only 408 requests for its llms.txt file. This means the site received only 408 requests out of 500 million visits, meaning the file was ignored by more than 1.2 million bots for every 1 request it received.
An Ahrefs study of 137,000 sites confirmed this lack of adoption, finding that 97% of existing llms.txt files received zero traffic in a month. This data suggests that while the file is “cheap” to create, it currently functions as a non-functional shortcut for site owners. Why is the advice everywhere then? It often stems from a desire to appear “future-proof” rather than a response to actual bot behavior.
Prioritizing on-page signals for AI retrieval
AI systems prioritize the same signals that drive traditional search results, such as clear headings and direct answers. Consistent naming for entities and schema markup that matches the visible page content are the primary ways to ensure a site appears in AI Overviews. The work required to optimize these elements is the only guaranteed way to improve LLM visibility today.
What is schema markup?
Schema markup is a standardized vocabulary of tags used to give search engines and AI models explicit information about a page’s content. It provides structured data that tells a bot exactly what a product, person, or event is. For AI search, schema ensures that the data the model retrieves matches the actual facts on the page.
The llms.txt proposal remains a speculative tactic until a major provider like OpenAI or Google announces they have updated their models to read it. Until that happens, maintaining a parallel markdown site creates an unnecessary maintenance overhead. If a major provider confirms support, the index file can be added in a few minutes, but the full site mirror should remain on hold.
What OpenAI, Google, and Anthropic have not yet confirmed about llms.txt
- OpenAI has not announced a specific integration date for the llms.txt proposal.
- Google has not confirmed whether their AI Overviews will prioritize.md files over standard HTML.
- Anthropic has not stated whether their crawler will recognize the llms.txt root directory.
A Site Owner’s Decision on AI Future-Proofing
Suppose Sarah is an SEO manager for a travel site with 500 blog posts. She wants to ensure her content is easily parsed by ChatGPT and Google AI Overviews. Sarah initially considers building a full markdown mirror of her site to satisfy the llms.txt proposal, but she calculates the maintenance cost of updating 500 files every time a post changes. Instead, she decides to audit her existing pages to ensure every post has a clear H1 tag, a direct answer in the first paragraph, and accurate schema markup. This allows her to optimize for current AI results without the overhead of a secondary site version.
If Google were to announce tomorrow that they now use llms.txt to prioritize content, Sarah’s decision would flip. In that scenario, she would immediately implement the root index file to gain a potential ranking advantage, but she would still likely delay the full markdown mirror until the benefits were proven.
Frequently Asked Questions
Does llms.txt block AI crawlers?
The llms.txt file does not block or allow AI crawlers. The proposal itself states that robots.txt remains the correct mechanism for controlling bot access via user-agent directives.
Do Google or OpenAI use llms.txt currently?
Google and OpenAI do not currently use the llms.txt file. John Mueller confirmed in 2025 that no AI system currently uses the file for content indexing.
How much work is required to fully implement the llms.txt proposal?
The llms.txt proposal requires creating a root index file and a markdown copy of every important page on the site. This requires maintaining a second version of the entire site in plain text alongside the real one.
Is llms.txt the same as robots.txt?
The llms.txt file is a proposed content index for LLMs, whereas robots.txt is the standard for blocking or allowing bots. Robots.txt provides the actual control for AI crawler instructions that llms.txt does not offer.
Details first shown in You Don't Need An LLMs.txt File (Yet) | SEO Walks Ep. 32 by The SEO Pub.