llms.txt: Does It Actually Help with GEO, or Is It Just Another SEO Trend?

Every time something changes in search, a new list of supposedly essential website optimizations seems to appear almost immediately.

Some eventually become established best practices. Others disappear within a few months, often after being promoted as something no business could afford to ignore.

With the rise of Generative Engine Optimization, we are seeing much the same pattern. One particularly interesting example is llms.txt.

The file is frequently described as a kind of robots.txt for artificial intelligence: something you add to your website to help ChatGPT, Gemini, Claude and Perplexity understand your content and, ideally, cite it in their answers.

The idea is interesting. The problem is that three very different claims are being treated as though they were interchangeable: that the file could be useful to an AI system, that major AI systems actually read it, and that publishing it improves a website’s visibility.

The first is plausible. The second needs evidence. The third, based on what we can document today, remains unproven.

Let’s look at why.

What is llms.txt, and where did it come from?

llms.txt is a proposal introduced by Jeremy Howard in September 2024, designed to make website information more accessible to large language models.

The original reasoning is fairly straightforward.

Web pages are primarily designed for people using browsers, whereas an automated system may only need to extract the information contained within them.

A typical HTML page can include menus, visual components, scripts, interactive elements, advertising, banners and plenty of other material that is not necessarily useful when answering a question.

For a system retrieving technical documentation or navigating a website’s resources, having access to a concise, structured, machine-readable overview could be convenient.

This is the thinking behind publishing a text file, generally located at:

https://www.example.com/llms.txt

Written in Markdown, it can contain a short introduction to the website, descriptions of its resources and a curated collection of links to important pages.

The format is not particularly complicated. Simplicity is one of its main strengths.

A practical example

Imagine a company developing hotel management software.

Its file might look something like this:

# HotelSuite

> HotelSuite develops property management
> software for independent hotels.

Official product documentation, features,
integrations and information for evaluating
the platform.

## Product

- [Features](https://example.com/features):
  Core property management functionality.

- [Integrations](https://example.com/integrations):
  Supported systems and external services.

- [Pricing](https://example.com/pricing):
  Available plans and pricing conditions.

## Technical documentation

- [API](https://example.com/docs/api):
  Official API documentation.

- [Getting started](https://example.com/docs/start):
  Initial configuration and setup.

## Optional

- [Blog](https://example.com/blog):
  Hospitality industry insights.

These URLs are illustrative.

Unlike an XML sitemap, the intention is not necessarily to list every page on the website. Instead, the file provides a curated selection of relevant resources, accompanied by short descriptions.

That distinction helps explain where the proposal might be useful.

If a software agent needs to understand a product and decides to consult this file, it can quickly obtain an organized overview of the main resources.

But there is an essential condition: the agent must actually choose to read it.

This is where things become much less straightforward when we move from a technical proposal to promises of improved visibility.

llms.txt is not the new robots.txt

Comparisons between the two files are common, but I find them misleading.

robots.txt is part of a standardized protocol used to communicate crawling rules to compliant automated agents.

It is a well-established mechanism supported by numerous crawlers and formally defined through a standard.

llms.txt serves a different purpose.

It was not designed to grant or deny access to particular pages. It proposes a collection of descriptions and links that compatible AI systems may decide to use.

The differences are worth making explicit.

File	Primary purpose
`robots.txt`	Communicate crawling rules to compliant bots
`sitemap.xml`	Identify URLs relevant to content discovery
`llms.txt`	Provide a curated guide to content for compatible systems

All three can coexist, but they are not interchangeable.

In particular, llms.txt is not an access-control standard and does not require AI systems to consult it.

Publishing the file does not grant crawling permissions, and omitting it does not prevent an AI assistant from finding website information through other channels.

Does Google use llms.txt?

On this point, the available information is fairly clear.

At a Google Search Central event in 2025, Gary Illyes explained that established SEO practices remained relevant for Google’s AI search features, without the need to adopt llms.txt.

In May 2026, Google published further guidance on optimizing for generative features in Search. In June, it explicitly clarified that Google Search does not use these files as a special signal, and that publishing one neither improves nor harms visibility or rankings in Search.

An important distinction is being made here.

Google may technically discover or index various file formats, but that does not mean it assigns them a specific role when selecting sources for AI Overviews or AI Mode.

So if someone claims that installing llms.txt is necessary to appear in Google’s AI-generated answers, they should be able to provide documentation contradicting Google’s own guidance.

There is an additional detail that can make the situation appear confusing.

In May 2026, Google’s experimental Lighthouse audits related to agentic browsing began checking for the presence of an llms.txt file.

I do not see this as a contradiction.

Lighthouse can consider the file a potential aid for agents navigating and interacting with websites, while Google Search does not use it as part of its ranking or AI Search source-selection process.

These are different products serving different purposes.

The fact that an auditing tool checks whether a file exists is not evidence that the file improves visibility in AI-generated answers.

What about ChatGPT, Claude and Perplexity?

The situation is less clear here, partly because not every detail of these systems’ internal processes is publicly available.

OpenAI documents different crawlers for different purposes, including OAI-SearchBot, associated with search functionality, and GPTBot, used to collect content for potential model training.

Perplexity and Anthropic also document automated agents used to access online resources.

However, the existence of a crawler capable of reading text files does not mean that it systematically requests llms.txt, or that it relies on the file to decide which sources to cite.

To evaluate that possibility, we need observations rather than assumptions about what is technically feasible.

Fortunately, some useful data became available in 2026.

What the data shows: a study of 137,000 websites

In June 2026, Ahrefs published an analysis based on approximately 137,000 domains monitored through its analytics tools.

The researchers examined which websites published an llms.txt file and, more importantly, whether those files actually received requests.

The findings were striking.

Around 28% of domains in the sample published an llms.txt file. Yet during May 2026, 97% of the identified files received no requests at all.

In other words, for the overwhelming majority of the websites observed, nobody requested the file during the period studied.

Among files that did receive requests, some traffic came from AI crawlers, but other requests originated from auditing tools, validation services and automated scanners.

This does not prove that llms.txt is useless under every possible circumstance.

First, the sample is not necessarily representative of the entire web. Websites using Ahrefs analytics may differ from an average website.

Second, one month of observations cannot establish that no system will ever use the format.

But there is one conclusion we can draw without overstating the results: publishing a file does not mean anyone is going to read it.

And if nobody requests the file, it is difficult to argue that it is directly improving a website’s visibility through that mechanism.

Another study: 83 websites monitored for 12 weeks

A separate study published by EZY Research on July 27, 2026 examined server logs from 83 websites publishing llms.txt, covering the period from April 27 to July 19.

The comparison between requests for robots.txt and llms.txt is particularly revealing.

Crawler	robots.txt requests	llms.txt requests
GPTBot (OpenAI)	3,990	7
ClaudeBot	3,120	9
PerplexityBot	775	0

The same study reported a different pattern for a Meta automated agent, which made substantially more requests to llms.txt.

These numbers should not be interpreted as describing the overall behavior of every AI application. They are observations from a specific group of websites, crawlers and dates.

Nevertheless, they illustrate an important distinction: a system may be technically capable of using a format without adopting it as a routine method for retrieving information.

And the difference in request volumes between robots.txt and llms.txt is substantial.

Even if the file is read, does it generate more citations?

This is the question I consider most important.

Suppose a crawler requests our website’s llms.txt file.

What have we demonstrated?

Only that the file was requested.

We do not necessarily know whether it was parsed, whether its links were followed, whether the information was used to generate an answer or whether it contributed to a citation.

To demonstrate a genuine GEO benefit, we would need to connect several observations:

A system actually accesses the file.
It retrieves one or more of the listed resources.
It uses the information in a generated answer.
The presence of the file produces a better outcome than a comparable situation without it.

The first two steps can be partially observed through server logs and other measurements.

The third can be investigated by examining responses and citations.

The fourth is the most difficult because it requires a comparison that accounts for other variables.

If a website publishes llms.txt and begins appearing more frequently in ChatGPT answers a month later, we cannot automatically conclude that the file caused the improvement.

During that period, the website’s content, external references, retrieval indexes or the behavior of the AI system itself may have changed.

That is why I am skeptical of any service promising increased AI citations simply by installing an llms.txt file.

When might llms.txt actually be useful?

So far I have focused on its limitations, but I do not want to reach the opposite extreme and dismiss the idea entirely.

There are situations where it makes sense.

Technical documentation and APIs

This is probably the use case closest to the original proposal.

A software project may have hundreds of pages describing configuration options, API methods, parameters, code examples and software versions.

An agent working with that documentation could benefit from a concise index, provided it has been designed to consult one.

The objective here is not to make ChatGPT mention the brand in a commercial recommendation. It is to help a compatible system locate the information required to complete a task.

That is a fundamentally different goal.

AI applications and agents you control

Imagine developing an AI assistant that regularly consults documentation from a collection of partner websites.

If we decide to use llms.txt as the entry point to those resources, we know the file will actually be read because we have implemented that behavior ourselves.

In this scenario, the file becomes part of a defined architecture rather than a bet on how external services might work.

We can assess its usefulness by measuring retrieval quality, answer accuracy and the operational cost of processing information.

Corporate websites and editorial projects

Here I would be considerably more cautious.

For a typical service business, e-commerce website or blog, I do not see enough evidence to assign llms.txt a high priority.

It can be a relatively simple addition, particularly when generated automatically from existing content.

But even an easy-to-create file introduces maintenance overhead. When important URLs, services or documents change, the information in the file needs to remain consistent.

An outdated content index can become less useful than a properly structured website navigation system.

How I would implement it if I decided to test it

If a client wanted to experiment with llms.txt, I would have no particular objection, provided expectations were realistic.

I would follow a few principles.

First, curate resources instead of copying the sitemap.

An indiscriminate list of thousands of URLs is unlikely to be a useful content guide. I would select important resources and organize them by subject and purpose.

Second, write precise descriptions.

A description such as “the most innovative solution for every business” tells us very little. It is much more helpful to explain what a page contains, who it is intended for and which questions it can help answer.

Third, avoid unnecessary duplication.

The file should direct systems toward canonical, up-to-date resources rather than introducing alternative versions that may eventually become inconsistent.

Fourth, automate maintenance where possible.

For complex websites, an index generated directly from the CMS or official documentation may be more reliable than a manually maintained file.

Fifth, do not confuse content guidance with crawler permissions.

I would continue using the appropriate mechanisms to manage crawler access, indexability and related directives rather than attempting to delegate those functions to llms.txt.

Finally, I would define the experiment in advance: what are we trying to establish, and what evidence would justify continuing to maintain the file?

How to check whether anyone reads the file

This is a relatively straightforward check if you have access to your server or CDN logs.

The starting point is to identify requests to /llms.txt and record at least the date, user agent, IP address, HTTP status and request volume.

For a typical Apache access log, a simple search might look like this:

grep -F 'GET /llms.txt ' access.log

This is only an example. The command should be adapted to the actual log format, HTTP methods and possible query parameters.

I would not rely solely on user-agent strings to identify crawlers, since those can be spoofed. Where possible, I would verify requests using the IP ranges and identification procedures documented by the providers.

I would also distinguish between search crawlers, training crawlers, agents acting on behalf of users and tools that merely check whether the file exists.

These represent different types of activity, and combining them into a single metric can be misleading.

After several weeks, we should be able to answer a practical question: is anyone actually requesting the file?

If not, I would avoid attributing any observed visibility improvements to it.

If the file is being requested, I would investigate which systems are accessing it and which resources they subsequently retrieve, within the limits of what server logs can reveal.

Even then, I would not claim to have reconstructed the internal process by which a language model selects sources for an answer.

Where I would invest before working on llms.txt

If I had to allocate a GEO project’s resources, I would start with activities that deliver clearer and more measurable benefits.

First, I would make sure important pages are accessible to the crawlers we want to reach, with no unintended technical indexing issues.

Next, I would focus on the content itself.

A company should explain precisely what it offers, who it serves, what its products or services actually do, what limitations they have and how they differ from alternatives.

Important information should be easy to locate, properly connected and consistent across pages.

I would also assess the brand’s presence outside its own website: documentation, editorial mentions, reviews and trustworthy independent sources.

Finally, I would establish a repeatable process for observing visibility in AI-generated answers, distinguishing brand mentions from actual citations and business outcomes.

All of these activities require more effort than uploading a text file.

But they also have an important advantage: they improve the website and the quality of its information regardless of whether a particular technical proposal is eventually adopted by major AI platforms.

So, should you implement llms.txt or not?

The answer depends on the project, not on the latest trend.

For a website with extensive technical documentation, or an AI application explicitly designed to consume the file, it may be a reasonable choice.

For a typical business website, I would consider it a low-priority experiment, provided implementation and maintenance costs remain limited.

I would not present it as a requirement for appearing in ChatGPT, and certainly not as a way to improve rankings in Google’s AI Search features, given Google’s explicit statements on the subject.

And I would never sell visibility improvements without establishing how they could be measured.

Ultimately, this debate illustrates a recurring problem in digital marketing: a technically possible solution is not necessarily an adopted solution, and an adopted solution is not necessarily an effective one.

My approach is the same as with any other SEO or optimization activity: understand the problem we are trying to solve, verify how the technology actually behaves and evaluate the results using observable evidence.

For now, llms.txt remains an interesting proposal with some genuine potential applications, but it is not a proven shortcut to generative search visibility.

And that distinction may be more valuable than any new file we decide to upload to a server.

References and further reading

Jeremy Howard, The /llms.txt file, original proposal, September 2024. https://llmstxt.org/
Google Search Central, Optimizing your website for generative AI features on Google Search, May 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
Search Engine Land, Google says llms.txt files won’t harm or help your search rankings, June 15, 2026. https://searchengineland.com/google-says-llms-txt-files-wont-harm-or-help-your-search-rankings-480264
Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, June 15, 2026. https://ahrefs.com/blog/llmstxt-study/
EZY Research, We Put llms.txt on 83 Websites. OpenAI Read It 7 Times, July 27, 2026. https://www.ezy.ai/research/do-ai-bots-read-llms-txt
OpenAI, Overview of OpenAI Crawlers. https://platform.openai.com/docs/bots
Search Engine Land, Google adds llms.txt check to Chrome Lighthouse, May 20, 2026. https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246