What Is llms.txt? A KWT Spider Guide to the New File Everyone's Talking About

What is llms.txt and do you really need one for your site?

If you have been around an SEO forum or Twitter/X thread at all in the last year, you’ve likely seen someone talking about llms.txt like it’s the new robots.txt. Some people say you can’t do without it. Some think it’s a solution looking for a problem. Honestly, both of these reactions are a little premature — so let’s slow down and actually look at what it is, where it came from, and if it’s worth your time right now.

The short version

llms.txt is a plain text file located at the root of your website (yoursite.com/llms.txt) intended to provide AI models and their crawlers with a neat, curated overview of your site — what it is, what’s on it, and which pages are most important. Think of it less as robots.txt which is about permissions – “you may or may not crawl this”, and more as a very condensed table of contents, written for a language model instead of a human visitor.

The idea was proposed in late 2024 by Jeremy Howard (co-founder of Answer.AI), as a way to solve a real problem: modern websites are messy for LLMs to parse. Navigation menus, ads, cookie banners, javascript rendered content – all that is noise for a model trying to quickly understand what a page is actually about. llms.txt cuts thru that and just gives the essentials in Markdown.

What really goes in one

The most common llms.txt file is very small, often less than a page. Typically includes: • An H1 with the name of your site or project

1. One line summary of what the site does
2. A handful of links to your most important pages - docs, key products, about page, etc with a brief description of each
3. "Optional" section sometimes for secondary pages which are useful but not necessary

There’s also a proposed llms-full.txt variant, a longer, more complete dump of your site’s content in Markdown — useful for sites like documentation sites where you want to hand over a lot more than a table of contents.

Is it actually doing anything yet?

Here’s the honest slightly unsatisfying part: there’s no evidence that major AI programs are reading llms.txt right now and using it to change how they cite or summarize your site. It’s not an official standard. This is not supported by OpenAI, Google or Anthropic in any official capacity. It's a convention that is emerging like how one company created sitemap.xml before it was adopted by search engines.

Not that it's bad. Some reasons why people are adopting it anyway:

1. It costs practically nothing to make. This is a one-off file not an ongoing commitment as much SEO work is.
2. It’s a bet on the method of travel. If AI crawlers do begin to prioritize sites with structured, machine-readable summaries – as they eventually did with sitemaps and structured data – having one in place already means you’re not scrambling later.
3. The exercise in itself is valuable. Even if no crawler ever reads the file it’s a good forcing function to write a tight accurate summary of what your site actually offers. It’s the same value you’d get writing a good meta description just at the site level versus the page level.

The honest caveat: Don’t treat this as a guaranteed ranking or visibility lever. Consider this a cheap, forward-looking hygiene – more “nice to have” than “must have,” at least for now.

How to make a real one?

You don’t need a tool to write a llms.txt file, it’s just Markdown in a text file. The general procedure:

1. Identify your most important pages. Not all of them, just the ones you'd want an AI system to get a sense of your site from if it could only read one file.
2. Write one clear sentence per page on what it is about. Avoid marketing language. Be literal and descriptive.
3. Save it as llms.txt in the root of your site, so it is available at yoursite.com/llms.txt.
4. Keep it up to date as your site’s key pages change — an outdated llms.txt is arguably worse than none at all, since it actively points to stale or wrong information.

If you’re auditing a larger site and want to auto-generate a starting draft instead of writing it by hand, this is one of the small utilities we added into KWT Spider 2.0. It can generate a baseline llms.txt from your site’s existing structure (titles, key pages, meta descriptions) that you then edit down into something accurate. It’s a starting point, not a final product; no automated tool knows which 5 pages actually matter to your business better than you do.

Want to auto-generate your first llms.txt? Try This SEO Crawler

If you’ve got ten minutes and a reasonable idea of the most important pages on your site, yes — it’s so cheap there’s not much reason not to have one. If you’re wondering whether to invest real time and resources on it, instead of say fixing real crawl errors or broken internal links, fix the fixes. llms.txt is a bet on the future, not a solution to problems you have today.