An llms.txt file is a short Markdown file that gives AI tools a map of your website. It’s often described as a “robots.txt for AI”, but it does something quite different, and its benefits are often overstated. This guide explains the format, what Google says about it, and when it’s worth adding. Make one with the llms.txt generator.

Quick answer: llms.txt is a proposed standard, published by Jeremy Howard in September 2024, for a Markdown file at /llms.txt that summarises a site and links to its key pages, so AI agents can find the right content without reading every page. Google says Google Search, including AI Overviews and AI Mode, doesn’t use it. It can still help AI agents and coding tools that look for it.

What llms.txt is for

Web pages are full of menus, scripts and adverts, which waste the limited context an AI model can read at once. The proposal at llmstxt.org suggests a single file that gives agents a short, curated overview of a site and points them to the pages that matter, ideally in clean Markdown.

It isn’t a permissions file. robots.txt tells crawlers what they may access. llms.txt only suggests what to read. To block AI crawlers, you still need robots.txt.

The format

The file is Markdown, and the sections go in this order:

  1. An H1 with the name of the site or project. This is the only required part.
  2. A blockquote with a short summary.
  3. Optional paragraphs or lists with more detail, but no headings.
  4. H2 sections containing lists of links, written as [name](url), each optionally followed by a colon and a note.
  5. An optional section headed Optional, for secondary links that agents can skip if they’re short on space.

An example

# Example Shop

> Example Shop sells handmade leather wallets and bags, made in Manchester and shipped across the UK.

All products come with a 2-year guarantee. Prices include UK VAT.

## Key pages

- [Shop all wallets](https://example.com/wallets/): Full range with prices
- [Delivery and returns](https://example.com/delivery/): UK and international shipping times
- [Care guide](https://example.com/care/): How to look after leather

## Optional

- [About us](https://example.com/about/)
- [Blog](https://example.com/blog/)

The file goes at the root of the site, such as https://example.com/llms.txt, or under a subpath like /docs/llms.txt to cover just that section. The proposal also suggests offering Markdown versions of important pages at the same URL with .md added.

Does Google use llms.txt?

No. Google’s guide to optimising for generative AI features on Google Search, published in May 2026, says you don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and that Google Search itself doesn’t use them. That includes AI Overviews and AI Mode. Google’s own summary is that optimising for generative AI search is still SEO.

Having the file doesn’t harm you either. Google’s guide says making these files for other services is fine.

Who does read it?

Site owners who have checked their server logs report that AI crawlers rarely request the file, so don’t expect it to bring traffic.

Should you add one?

Your site Worth it?
Software documentation or an API Yes. Agents and coding tools are the main users
A product, SaaS or tool site Maybe. It’s quick to make and does no harm
A blog or local business site Low priority. Spend the time on helpful content and good SEO

If you add one, keep it short, link only to pages you’d want an assistant to quote, and update it when key pages change. Make it in a minute with the llms.txt generator, and control which AI crawlers can access your site with the AI crawler robots.txt generator. More tools are in the AI tools guide.

llms.txt is a proposal, not an official web standard. Support from AI services may change.

Frequently asked questions

What is llms.txt?

A proposed standard for a Markdown file at the root of a website that summarises the site and links to its key pages, to help AI agents find the right content.

Does llms.txt help with Google rankings or AI Overviews?

No. Google says Google Search, including AI Overviews and AI Mode, doesn’t use llms.txt or other special AI files.

Is llms.txt the same as robots.txt?

No. robots.txt controls which URLs crawlers may access. llms.txt only suggests which pages to read. Use robots.txt to block AI crawlers.

Where do I put llms.txt?

At the root of your site, such as example.com/llms.txt, or under a subpath like /docs/llms.txt for one section.

What is the only required part of llms.txt?

An H1 heading with the name of the site or project. A summary and lists of links are recommended.

Who uses llms.txt?

Mainly AI coding tools and agents working with documentation sites. AI search crawlers rarely request it.

Sources

Last updated: