What is llms.txt?
llms.txt is a proposed standard (analogous to robots.txt) for a markdown file at a website's root that gives large language models a structured summary of the site's content and key pages. The proposal was introduced by Jeremy Howard in September 2024 at llmstxt.org.
What to know in practice
- Format: H1 with site name, blockquote summary, H2 sections listing key pages with descriptions, followed by optional H2 'Optional' section with deeper material.
- Adoption is still mostly opt-in: Anthropic's tooling reads it, but ChatGPT crawlers and Google don't yet (as of mid-2026).
- Companion file llms-full.txt expands the same structure with full body text — useful when an LLM needs the actual content, not just navigation.
- Practical value today: modest. Cost to deploy: trivial (a single text file). Recommended as a low-effort optimization even though impact is currently limited.
Common misconception
llms.txt is NOT a way to prevent LLMs from crawling your site — that's robots.txt with bot-specific disallow rules (GPTBot, ClaudeBot, PerplexityBot, etc.). llms.txt is the opposite: it INVITES LLMs to consume a structured summary.
Related terms
- Answer Engine Optimization (AEO) → SEO & AEO