Llms.txt is a special text file that allows websites to be understood more effectively by artificial intelligence systems and large language models. The file is placed in the root directory of the website and helps AI systems like ChatGPT, Google Gemini, Claude, and Perplexity process content more accurately.
Origin and Purpose
The llms.txt format was proposed by Jeremy Howard in September 2024 as a solution to the problem of HTML structure complexity for AI systems. Web content often contains complex structures, navigation menus, advertisements, and JavaScript, which makes it difficult for language models to understand the content.
Key Differences from robots.txt
Llms.txt:
- Helps large language models better understand website content
- Presents important content in a simple format for AI systems
- Goal: optimization for generative AI (GEO – Generative Engine Optimization)
- Written in Markdown format
Robots.txt:
- Controls how search engine bots crawl the site
- Allows or prevents crawling of certain pages
- Goal: traditional SEO optimization for search engines
Structure of llms.txt file
The file should contain:
- H1 header with project/site name
- Brief description block of the project
- Detailed information in paragraphs or lists
- Link lists to relevant documents and resources
How to Integrate
- Prepare the file in Markdown format
- Upload to the root directory of the website
- Add a reference in robots.txt
# LLMs.txt files for AI systems
AI-crawlers: /llms.txt
AI-crawlers-full: /llms-full.txt
LLMs-txt: https://alimbekov.com/llms.txt
LLMs-full-txt: https://alimbekov.com/llms-full.txt
- Check file accessibility
- Configure necessary permissions for AI bots
- Regularly update content
The article emphasizes the importance of monitoring requests from AI systems after implementing the file to assess its effectiveness and understand how the information is being used by different AI bots.
View files: