llms.txt and SEO: what it can and cannot do

llms.txt is a proposed Markdown file, usually placed at a site’s root, that curates important resources for language-model use. It is not a web standard, crawler permission system, XML sitemap replacement, or confirmed ranking factor. Publishing one can be a reasonable low-cost experiment for a documentation-heavy site, but it should sit behind higher-confidence SEO and content work.

The distinction matters because vendors often present a simple file as an “AI visibility” requirement. Current official platform documentation does not support that claim.

Understand what the llms.txt proposal asks for

The llms.txt proposal describes a root-level /llms.txt file written in a specific Markdown-oriented format. It suggests a project name, a short summary, optional detail, and curated link sections. The proposal’s goal is to help language models find concise, useful context at inference time.

That is a proposal maintained by its advocates. It is not an Internet Engineering Task Force standard or a directive that every model provider has agreed to honor. A valid file proves only that you published a file matching the proposal. It does not prove that a crawler fetched it, that a retrieval system used it, or that a generated answer cited any included page.

This does not make the idea useless. A concise machine- and human-readable map may be helpful for developer tools or agents that intentionally support it. The right posture is optional interface, measured adoption—not mandatory ranking infrastructure.

Keep llms.txt, robots.txt, and sitemaps separate

These files have different purposes and should not replace one another.

Google documents robots.txt behavior in its robots.txt specification guide. The XML sitemap format is documented at the official Sitemaps protocol site. These protocols have established consumers and defined purposes. Even then, permission and discovery are not guarantees of indexing.

Do not place crawler rules in llms.txt. Do not remove canonical URLs from navigation because they appear in the file. Do not stop maintaining XML sitemaps.

Check platform guidance before making claims

Google says no special AI text file is required for sites to appear in its AI features. Its official AI features guidance directs site owners back to established SEO fundamentals.

OpenAI publishes crawler names and access controls in its bot documentation. That documentation is the appropriate source for decisions about OpenAI crawler access. It does not state that llms.txt guarantees inclusion or citation.

These are verified platform facts as of the article date. Platform behavior and documentation can change, so review the source before briefing leadership. Claims such as “ChatGPT requires llms.txt” or “Google rewards llms.txt” need direct platform evidence; without it, they are vendor hypotheses or misinformation.

For a wider review of what AI visibility work rests on documented controls, see Leaf’s AEO vs SEO guide.

Decide whether the experiment deserves priority

An llms.txt experiment has a low implementation cost only when the site already has clean information architecture and maintained canonical resources. Otherwise, the file becomes another stale index pointing to weak pages.

Use this priority table:

The opportunity cost is not the time required to create version one. It is maintenance. Every renamed product, archived guide, pricing change, or documentation migration can make the file less trustworthy.

Design a bounded llms.txt file

If the experiment clears the priority test, keep the file selective. Include the official company or project description, main product or service pages, documentation entry points, policy pages needed to interpret the service, and a small number of genuinely authoritative resources.

Avoid dumping the entire sitemap into Markdown. A curated file that lists thousands of URLs has abandoned the proposal’s navigational purpose. Avoid promotional adjectives, unsupported market claims, duplicate links, tracking parameters, and temporary campaign pages.

A practical B2B structure might contain:

# Acme Data Platform

> Acme provides managed data-pipeline monitoring for B2B engineering teams.

Canonical product, documentation, and policy resources.

## Product

- [Platform overview](https://example.com/platform): Supported workflows and deployment options.
- [Pricing](https://example.com/pricing): Current plans, usage units, and exclusions.

## Documentation

- [Getting started](https://docs.example.com/start): Installation and first monitored pipeline.
- [Authentication](https://docs.example.com/auth): SSO, SCIM, API keys, and limitations.

## Policies

- [Security](https://example.com/security): Security controls and assurance documents.

This is an illustrative artifact, not evidence that any platform will consume it. The descriptions should match visible destination content.

Run an implementation and monitoring checklist

Treat the file like production content:

• □ Serve /llms.txt with a successful response and plain-text-compatible content type.
• □ Use absolute canonical URLs with no tracking parameters.
• □ Confirm every listed URL resolves to the intended public resource.
• □ Keep company and product descriptions consistent with canonical pages.
• □ Exclude private, paywalled, personalized, or legally restricted material.
• □ Preserve existing robots.txt and XML sitemap controls.
• □ Assign an owner and review date.
• □ Add link checking to an existing content-maintenance process if feasible.
• □ Record server requests separately from downstream citations.
• □ Define a removal condition if the file becomes stale or unused.

Where logs are available, look for requests to /llms.txt, but interpret them cautiously. User agents can be spoofed, and a request does not establish that a model used the content. If a tool explicitly documents support, test that tool’s behavior according to its documentation.

Measure an experiment without inventing ROI

Write the hypothesis before launch: “Publishing a curated llms.txt may help supporting tools discover the canonical documentation set.” Then record the file version, launch date, request logs, documented consumers, and any observable referral or citation changes.

Do not claim success because the URL was crawled once. Do not attribute a new ChatGPT citation to the file without a design that supports that conclusion. Generated answers vary, and many other source changes can occur during the same period.

If AI visibility is the actual goal, pair the experiment with a fixed prompt test and source review. Leaf’s guide to ranking in ChatGPT without the ranking myth explains how to define that panel. A broader AEO checklist will identify higher-confidence access, evidence, and identity work.

Put llms.txt in the right place on the roadmap

A sensible order is: fix crawler and index controls; establish canonical product facts; improve useful content and internal links; validate structured data where relevant; set a repeatable visibility baseline; then test optional interfaces such as llms.txt.

That order is not anti-experiment. It protects the experiment from being asked to solve problems it was never designed to solve. A map cannot repair the destination.

Publish llms.txt if the audience or tools you care about support it, the maintenance cost is acceptable, and expectations are explicit. Otherwise, spend the same hour correcting a source page that buyers and search systems already use.