llms.txt vs robots.txt
People call llms.txt the robots.txt for AI, and it is not. robots.txt tells bots which pages they may not fetch. llms.txt is a short guide that points AI tools to the pages worth reading. It blocks nothing at all. To keep an AI bot out, you still need a line in robots.txt.
llms.txt
A Markdown file at a site's root. It points AI tools to the pages that matter most.
llms.txt is a plain text file that gives AI tools a short guide to a website. It sits at the root of the site, the same place as robots.txt. It is written in Markdown: a title, a one-line summary, then lists of links to the most useful pages. It was proposed in September 2024 to help AI tools read a site. It is a proposal, not a standard, so no AI tool has to read it.
robots.txt
A text file that tells crawlers which parts of a site not to fetch.
robots.txt is a plain text file at the root of a site that tells crawlers which paths they may not fetch. Each rule names a crawler and a path to allow or block. Well-behaved crawlers obey it, but it is a request, not a lock. It controls crawling, not indexing. A blocked page can still show in results if other sites link to it.
In one line
- llms.txt
- A Markdown file at a site's root. It points AI tools to the pages that matter most.
- robots.txt
- A text file that tells crawlers which parts of a site not to fetch.