One-Sentence Summary
llms.txt is a Markdown file placed in a website's root directory that tells AI, in a simple format, "what this site is and which content is most worth reading" — key infrastructure for getting AI to cite your content first.
Background: The Clash Between AI Crawlers and Human-Facing Web Pages
Web pages are designed for humans: navigation, ads, pop-ups, JavaScript rendering — all of that is noise to AI. When an AI crawler fetches a page, extracting clean text from HTML is both difficult and imprecise, and with a limited context window, it cannot digest an entire website.
In September 2024, Jeremy Howard, co-founder of Answer.AI, formally proposed the llms.txt specification (the spec is hosted at llmstxt.org), using a single plain-Markdown file to tell AI models what the site is, which information matters most, and which pages to read first.
How llms.txt Differs from robots.txt and sitemap.xml
| File | Role | Analogy |
|---|---|---|
| robots.txt | Controls access permissions (what cannot be crawled) | Gate and access control |
| sitemap.xml | Lists all URLs (page discovery) | Floor plan |
| llms.txt | Guides understanding and priorities (what's worth reading) | Private guided tour |
robots.txt is a set of "prohibitions," and sitemap.xml is a "complete catalog"; llms.txt, by contrast, is a recommendation list — proactively telling AI what to read. It is an act of collaboration rather than control.
Format Specification (llmstxt.org v2)
A standard llms.txt contains, in order:
- H1 heading: the site or project name (the only required part).
- Blockquote: a one-sentence introduction containing the key information needed to understand the rest of the content.
- Plain paragraph: more detailed background (must not be a heading).
- H2-grouped link lists: Markdown links with short descriptions listed under each group.
Link format:
- [Link name](https://full-URL): a brief description of the linkAuthoring tips:
- Every link must be followed by a one-sentence description of what the page covers and whether it is worth citing.
- Sort by priority, putting the most important content first.
- Only include core pages that are worth citing; a longer list is not necessarily better.
Who Uses It
- AI vendors themselves: OpenAI, Anthropic, and Google Gemini have all published llms.txt for their developer documentation.
- Chrome Lighthouse: audits whether a site has llms.txt as part of its agentic browsing checks.
- Documentation platforms: documentation tools such as Mintlify automatically generate llms.txt for sites.
What This Site Does
This site uses vitepress-plugin-llms to auto-generate /llms.txt and /llms-full.txt at build time:
llms.txt: Markdown navigation of the site's core content (titles + summaries + links).llms-full.txt: a complete collection of every page's full Markdown content, letting AI obtain all core information in a single request.
You can visit this site's llms.txt directly to see it in action.
Verifying After Publication
- Visit
https://your-domain/llms.txtdirectly in a browser to confirm it opens properly. - Submit the site in Google Search Console and Bing Webmaster Tools.
- After some time, search this site's core keywords in DeepSeek, Doubao, ChatGPT, and other tools, and observe whether AI answers cite this site's content.
Sources and References
- llms.txt specification (v2): llmstxt.org
- Spec proposal repository (AnswerDotAI/llms-txt): github.com/AnswerDotAI/llms-txt
- Mintlify's official explanation of llms.txt: mintlify.com/docs/ai/llmstxt