The Definitive Guide

What Is llms.txt?

AI systems are reading the web differently than search crawlers ever did. llms.txt is a proposed convention for handing them a clean, curated map of your best content. Here's what it is, what it isn't, and whether you should publish one — with no hype about what engines actually support.

Explore AEO Services Try Our Robots.txt Generator

Quick Answer

llms.txt is a plain-Markdown file placed at your site root (yoursite.com/llms.txt) that gives large language models a curated index of your most important content — a short site summary plus organized links with descriptions. It was proposed by Jeremy Howard of Answer.AI in September 2024 as a community convention. Publisher adoption is growing, but no major AI engine has officially committed to consuming it — so treat it as low-cost preparation, not a proven visibility lever.

What Is llms.txt, Exactly?

Let's start with the problem it tries to solve. Modern web pages are built for humans and for Google's rendering pipeline — they carry navigation menus, cookie banners, JavaScript frameworks, ads, and layout markup. When a large language model fetches one of those pages, most of what it reads is noise. Context windows are finite, and every token spent parsing your mega-menu is a token not spent understanding your actual expertise.

llms.txt is a proposed answer to that problem. In September 2024, Jeremy Howard of Answer.AI published a specification suggesting that websites place a Markdown file at /llms.txt containing a concise, curated guide to the site: what it is, what it offers, and where its most important content lives. The format is deliberately human-readable — an H1 title, a blockquote summary, and sections of annotated links.

The proposal also defines an optional companion, /llms-full.txt, which concatenates the full text of your key pages into one clean document. Documentation-heavy companies have been the earliest adopters, because their content sets are well-defined and their users increasingly ask AI assistants questions that the docs answer.

Now for the part most articles gloss over: llms.txt is a convention, not a standard that engines have agreed to honor. Google representatives have publicly said Google does not use it. OpenAI, Anthropic, and Perplexity have not announced formal support. Publishing the file is a bit like leaving a well-organized welcome packet by the door — helpful if a visitor picks it up, but nobody is obligated to.

So why does it keep gaining traction? Because the cost of publishing one is nearly zero, the downside risk is nil, and the direction of travel is clear: more of the web's consumption is happening through AI intermediaries, agents, and on-demand fetchers that benefit from a clean content index today, regardless of what the big crawlers formally support. Early adopters are positioning themselves for a possible future standard while making life easier for the AI tools that already read sites on demand.

Honesty Check

No major search engine or AI platform has officially committed to consuming llms.txt. If a vendor tells you that publishing one will boost your rankings or guarantee AI citations, they are overclaiming. The honest case for llms.txt is low cost, zero risk, speculative upside — and that's a perfectly good reason to publish one, as long as you know what you're buying.

The llms.txt Format, Section by Section

The proposed spec keeps things intentionally minimal. A valid llms.txt file is plain Markdown with a defined structure:

H1 Title — The Site or Project Name

The file opens with a single H1 heading naming your site, company, or project. This is the only strictly required element. Example: "# Webvello". It anchors everything that follows so an LLM knows whose content it is reading.

Blockquote Summary — What This Site Is

Immediately after the title comes a Markdown blockquote with a short, factual summary of the site — one to three sentences covering what you do, who you serve, and what kind of content lives here. Write it as if a language model will quote it verbatim, because one might.

H2 Sections — Curated Link Lists

The body of the file consists of H2 sections ("Services," "Guides," "Docs," "Policies") each containing a bulleted list of Markdown links, with each link followed by a colon and a one-line description of what that page covers. This is where curation matters: link your best, most representative content — not everything.

The "Optional" Section — Skippable Extras

The spec reserves an H2 section literally named "Optional" for secondary resources. The convention: when an LLM is short on context space, it can skip everything under Optional and still get an accurate picture of the site. It's a built-in priority signal.

llms-full.txt — The Full-Content Companion

Optionally, sites can also publish /llms-full.txt containing the complete text of their core pages in one Markdown document. This lets an AI system ingest your entire knowledge base in a single fetch. It's most practical for documentation sites; for large marketing sites it can be unwieldy to maintain.

Pro Tip

Write your llms.txt descriptions the way you'd want an AI assistant to describe you to a customer. Specific beats promotional: "Guide to robots.txt rules for AI crawlers, with copy-paste examples" will serve you better than "Our amazing award-winning blog."

llms.txt vs. robots.txt vs. XML Sitemap

These three root-level files get confused constantly. They do very different jobs — and only two of them have established, widespread support:

Factor	robots.txt	XML Sitemap	llms.txt
Purpose	Control crawler access	List all indexable URLs	Curate key content for LLMs
Format	Plain-text directives	XML	Markdown
Audience	Search & AI crawlers	Search engine crawlers	LLMs, AI agents, on-demand fetchers
Content	Allow/Disallow rules	URLs + metadata, no content	Summary + annotated links (+ full text in llms-full.txt)
Selectivity	Rule-based	Exhaustive	Deliberately curated
Status	De facto standard since 1994	Standard since 2005, engine-supported	Community proposal (Sept 2024)
Engine support	Broad (with exceptions)	Official across search engines	No official commitments from major engines

The takeaway: robots.txt controls access, your sitemap declares existence, and llms.txt proposes guidance. Get the first two right before you touch the third — a well-configured robots.txt that makes deliberate decisions about AI crawlers matters far more today than any llms.txt file.

Get Your Crawler Files Right First

Before experimenting with llms.txt, make sure your robots.txt makes deliberate, correct decisions about GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Our free generator builds it for you.

Open the Robots.txt Generator Learn About AI Crawlers

Should You Publish an llms.txt File?

Here's the balanced view — the genuine reasons to do it, and the claims you should ignore:

Real: Near-Zero Cost, Zero Risk

An llms.txt file takes an hour or two to write, is invisible to normal visitors, and has no effect on your traditional SEO. There is no plausible downside to publishing an accurate one.

Real: Useful to On-Demand Fetchers Today

AI assistants and agents that fetch pages at a user's request — as opposed to bulk training crawlers — can use a clean content index right now. Some developer tools already read llms.txt files when exploring a site.

Real: Early Positioning If Support Formalizes

If any major platform adopts the convention, sites with existing, well-maintained llms.txt files benefit immediately. The proposal has visible momentum in the documentation and developer-tools world.

Real: A Forcing Function for Curation

Writing the file forces you to answer a useful question: "If an AI could only read ten pages of my site, which ten?" That exercise alone often surfaces gaps and priorities in your content strategy.

Overclaimed: "It Boosts Your Rankings"

There is no evidence llms.txt affects search rankings. Google has indicated it does not use the file. Anyone selling llms.txt as an SEO ranking tactic is selling speculation as fact.

Overclaimed: "AI Engines Will Cite You More"

AI citations are driven by content quality, structure, retrievability, and authority — the fundamentals of GEO. No engine has confirmed that llms.txt influences citation selection. Do the fundamentals first; treat llms.txt as a bonus.

How to Create an llms.txt File: 6 Steps

If you've decided the low-cost bet is worth it, here's how to do it properly:

Inventory Your Highest-Value Pages

List the 10-30 pages that best represent your expertise and answer your customers' most common questions: core service pages, flagship guides, pricing, policies, and about/contact pages. This is a curation exercise, not a site dump — if everything is important, nothing is. Your analytics and Search Console data will tell you which pages already earn attention.

Write the Title and Summary Blockquote

Open the file with an H1 containing your brand name, then a blockquote of one to three sentences stating plainly what you do, who you serve, and where you operate. Keep it factual — no superlatives you can't back up. This summary is the single most likely part of the file to be read and repeated by a language model.

Organize Links into Logical H2 Sections

Group your curated links under clear section headings: Services, Guides, Tools, Pricing, Policies. Under each heading, list Markdown links with a colon and a one-line description after each. Descriptions should say what a reader (human or machine) will actually find on the page — specific enough that an LLM can decide whether to fetch it.

Add an "Optional" Section for Secondary Content

Move nice-to-have resources — older posts, supplementary references, press pages — under an H2 literally titled "Optional." Per the proposed spec, this signals that these links can be skipped when an AI has limited context space. It's your built-in priority tiering.

Serve It at the Root as Plain Text

Publish the file at yoursite.com/llms.txt with a text-based content type, exactly like robots.txt. In most frameworks this means dropping llms.txt into your public or static directory. Verify it loads in a browser as raw Markdown — not wrapped in your site template — and that your robots.txt doesn't accidentally block it.

Maintain It Like You Maintain Your Sitemap

A stale llms.txt that links to dead pages or describes services you no longer offer is worse than none at all, because it feeds wrong information to any system that does read it. Add llms.txt review to the same checklist you use when launching, renaming, or retiring key pages.

Where llms.txt Fits in a GEO Strategy

llms.txt is one small tile in the larger mosaic of Generative Engine Optimization. The heavy lifting is still done by well-structured content, clear entity signals, schema markup, and crawlable pages. If you're prioritizing, do those first — then add llms.txt as the finishing touch.

Common llms.txt Mistakes (And How to Avoid Them)

The file is simple, but people still get it wrong in predictable ways:

Mistake: Dumping Every URL Into the File

The Fix: llms.txt is not a second sitemap. Its entire value is curation — a short, prioritized map of what matters. If your file lists 500 URLs with no descriptions, an LLM gains nothing over crawling your sitemap. Keep it to your genuinely important pages, each with a meaningful one-line description.

Mistake: Treating It as a Ranking Tactic

The Fix: Publishing llms.txt and expecting search or citation gains sets you up for disappointment — no engine has committed to using it. Set the expectation internally as "cheap future-proofing and agent hygiene," and invest your real optimization effort in content structure, schema, and authority.

Mistake: Letting It Go Stale

The Fix: A file describing services you discontinued or linking to 404s actively misinforms any system that reads it. Tie llms.txt updates to your content release process: when a key page launches, moves, or dies, the file gets reviewed in the same change.

Mistake: Writing Marketing Fluff Instead of Facts

The Fix: Language models summarize what you give them. If your descriptions read "the world's most innovative solution," that is either ignored or repeated verbatim in a way that erodes trust. Plain, specific, verifiable descriptions serve you better in every scenario.

Mistake: Confusing llms.txt With Crawler Control

The Fix: llms.txt cannot block anything. If you want to control which AI crawlers access your site — for training, search indexing, or both — that decision lives in robots.txt user-agent rules. Use our robots.txt generator to configure GPTBot, ClaudeBot, PerplexityBot, and Google-Extended deliberately, then use llms.txt for guidance on top.

The llms.txt Mindset

Think of llms.txt as documentation for machines about your website. Documentation is only valuable when it's accurate, curated, and maintained — and nobody promises to read it. Publish it because it's cheap and honest, keep it current, and let the ecosystem catch up.

Frequently Asked Questions About llms.txt

Straight answers about the proposed standard — including what engines do and don't support.

An llms.txt file is a plain-Markdown file placed at the root of a website (yoursite.com/llms.txt) that gives large language models a curated, structured overview of the site's most important content. It was proposed in September 2024 by Jeremy Howard of Answer.AI as a community convention — not an official standard from any search engine or AI company. The idea is simple: web pages are cluttered with navigation, scripts, and boilerplate that make them hard for LLMs to parse efficiently, so llms.txt offers a clean, curated map of what actually matters on your site.

As of now, no major AI platform has officially committed to honoring llms.txt. Google representatives have publicly indicated that Google does not use it, and OpenAI, Anthropic, and Perplexity have not announced formal support for the proposal either. That is the honest state of affairs. Adoption on the publisher side is growing — many documentation platforms and software companies publish llms.txt files — but publishing a file and having engines consume it are two different things. Treat llms.txt as a low-cost, forward-looking bet rather than a proven ranking or citation lever.

The llms.txt file is a curated index: a short site description followed by organized lists of links to your key pages, each with a one-line summary. The optional llms-full.txt companion file goes further — it contains the full text content of those pages concatenated into a single Markdown document, so an LLM can ingest your entire core content in one request without crawling page by page. Most sites start with llms.txt alone; llms-full.txt is most common for documentation sites where the complete content set is well-defined.

They solve opposite problems. robots.txt is a gatekeeping file — it tells crawlers which parts of your site they may or may not access, and it has decades of established support from search engines and AI crawlers alike. llms.txt is a guidance file — it tells LLMs which content is most important and how it is organized, assuming they choose to read it. robots.txt says "here is what you can touch"; llms.txt says "here is what you should read first." robots.txt compliance is widely (though not universally) respected; llms.txt consumption is still speculative.

An XML sitemap lists every indexable URL on your site in a machine format built for search engine crawlers — it is exhaustive, unprioritized, and contains no content. An llms.txt file is the opposite: selective, curated, human-readable Markdown that includes context about what each linked resource covers and why it matters. A sitemap answers "what URLs exist?" while llms.txt answers "what should an AI read to understand this site?" They complement each other rather than compete.

The proposed format is deliberately simple Markdown. It starts with an H1 heading containing the site or project name, followed by a blockquote with a short summary. After that come optional H2 sections (such as "Docs," "Guides," or "Policies"), each containing a bulleted list of links in the form of a Markdown hyperlink followed by a colon and a brief description. An optional "Optional" section lists secondary resources that an LLM can skip when context space is limited. Because it is plain Markdown, it is easy for both humans and machines to read.

For most sites, yes — with expectations set correctly. The cost is low: an hour or two to curate your most important pages into a Markdown file. The downside risk is essentially zero, since the file is invisible to normal visitors and does not affect traditional SEO. The upside is speculative but real: if AI platforms formalize support, early adopters will already be positioned, and some AI tools and agents that fetch pages on demand can already make use of a clean content index today. What you should not do is expect llms.txt alone to earn AI citations — content quality, structure, and authority still do that work.

There is no verified evidence that publishing an llms.txt file improves search rankings or AI citation rates today, and anyone claiming otherwise is ahead of the data. Traditional search engines rank pages based on their established signals, and AI answer engines retrieve content primarily through their own crawlers and search indexes. Where llms.txt may help right now is with user-initiated AI fetches and agentic tools that read a site on demand — a clean, curated index makes their job easier. Think of it as good hygiene for an AI-mediated web, not a growth tactic with measurable returns yet.

Include your highest-value, most representative content: a one-paragraph description of who you are and what you do, links to your core service or product pages, your best explanatory guides and documentation, pricing or policy pages that answer common questions, and contact information. Keep descriptions honest and specific — this file may literally be read verbatim by a language model summarizing your business. Leave out thin pages, promotional fluff, and anything you would not want quoted. Update the file when your key content changes, just as you would a sitemap.

Want to Be Visible Where AI Answers Are Written?

llms.txt is the easy part. Getting your content structured, retrievable, and authoritative enough for AI engines to actually cite — that's the real work. Our team builds AI-search visibility strategies grounded in what engines demonstrably do, not hype.

Book a Free Call View AEO Services

SEO Glossary·Robots.txt Generator·What Is GEO?·Best GEO Software