What is llms.txt?
llms.txt is a simple text file in Markdown format located in the root directory of a website — accessible at yourdomain.com/llms.txt. It contains structured information about the website, specifically prepared for Large Language Models (LLMs) like ChatGPT, Claude and Perplexity.
The idea behind it is simple: when an AI system crawls a website or needs to provide information about it, a compact well-structured overview helps deliver accurate and complete answers. Instead of the model laboriously piecing together information from dozens of subpages, it finds everything essential in llms.txt at a glance.
In short: llms.txt is a voluntary, machine-readable business card for your website — written specifically for AI systems. It answers the most important questions an LLM might have about your website before it starts crawling.
Where does the standard come from?
The llms.txt specification was proposed in 2024 by Jeremy Howard — co-founder of fast.ai and one of the most influential AI researchers of recent years. The idea quickly gained traction in the AI and web community, partly because Anthropic (the maker of Claude) actively supports the specification and provides an llms.txt file on their own website.
The standard is deliberately kept simple: no complex syntax, no new file format, no special parser needed. Plain Markdown that anyone can read and write — and that is simultaneously easy for machines to process.
Important to note: llms.txt is not yet an official W3C standard. It is a community standard that is establishing itself in practice — similar to how robots.txt worked in the 1990s before it was formalised.
llms.txt vs. robots.txt — the difference
| Feature | robots.txt | llms.txt |
|---|---|---|
| Purpose | Tells crawlers what NOT to crawl | Explains to AI what the website IS |
| Format | Custom keyword format | Markdown |
| Audience | All web crawlers | Specifically LLMs and AI systems |
| Content | Rules and restrictions | Descriptions and context |
| Standard | Official RFC standard | Community standard (2024) |
| Adoption | Almost every website | Still rare — growing |
The two files are not mutually exclusive — quite the opposite. robots.txt controls access, llms.txt provides context. A website can and should have both.
Who reads llms.txt?
The file is read by AI crawlers and LLM systems that actively look for it. Currently known supporters:
- Perplexity AI — actively reads llms.txt and uses the content for answers
- Anthropic / Claude — officially supports the standard, has its own llms.txt
- GPTBot — OpenAI has not yet officially confirmed the standard, but llms.txt is taken into account during processing
- Other AI tools — the ecosystem is growing, many smaller AI search engines already support llms.txt
Being early pays off: Currently fewer than 1% of all websites have an llms.txt. Anyone who creates one now has a clear advantage over competitors who are not yet providing AI-specific context.
Structure and syntax
An llms.txt file is pure Markdown. There is a recommended structure that most AI systems expect:
Required and optional fields
Strictly speaking there are no required fields — llms.txt is an open standard. Recommended at minimum however are: name/title of the website, a short description (About) and contact information. Everything else is optional but useful.
Ready-made examples for different website types
Example: SaaS tool / web application
Example: Local business
Example: Blog / content website
Step-by-step: create and deploy llms.txt
Step 1: Create the file
Create a new text file named llms.txt. Use UTF-8 encoding without BOM. Write the content in Markdown format following the template above — adapted to your website.
Tip: Don't want to write your llms.txt manually? The llms.txt Generator by llmshub.de automatically crawls your website and fills in all fields — you just review and download.
Step 2: Place it in the root directory
The file must be accessible at the main URL of the domain: https://yourdomain.com/llms.txt. For most web servers this means: place the file in the root directory of the web server — the same directory where index.html lives.
Step 3: Check the Content-Type
The web server should serve the file as text/plain. For .txt files this is normally automatic. To verify:
Step 4: Reference in robots.txt (optional but recommended)
To help AI crawlers reliably find your llms.txt, you can add a comment at the end of robots.txt:
Note: A standardised directive for llms.txt in robots.txt does not yet exist. A comment does no harm but is not evaluated by all crawlers. More important is that the file is accessible at the correct URL.
llms-full.txt — the extended variant
Alongside llms.txt there is an optional extension: llms-full.txt. This file contains the complete content of the most important pages in a form optimised for LLMs — as a single readable file.
The idea: instead of an LLM having to crawl dozens of subpages individually, it finds all relevant content compactly summarised in llms-full.txt. This is particularly useful for documentation sites, knowledge bases or extensive guides.
For most normal websites llms-full.txt is a nice-to-have, not a requirement. llms.txt is the more important first step.
Checklist
- llms.txt created in the root directory of the domain
- File accessible at yourdomain.com/llms.txt (HTTP 200)
- Content-Type is text/plain
- Minimum content: name, description, contact
- Markdown format correct (headings with ##)
- Saved as UTF-8 without BOM
- Reference added in robots.txt (optional)
- Content kept up to date regularly
Does your website have an llms.txt?
AI-Ready Check checks for free whether your llms.txt is present and correctly accessible — along with 20 other factors for AI visibility.
Test for free now →
Your site already has an llms.txt but you're not sure if it's technically and structurally correct?
→ Validate now with llmshub.de