llms.txt Setup Guide: robots.txt, Templates, and Impact

llms.txt is a Markdown file placed at a website's root that organizes key information so AI language models can quickly understand the site. It's a format proposed by Jeremy Howard in late 2024 ^[1], and although its name resembles robots.txt, its role is different. This post covers what llms.txt is, how much impact it really has, and how to set it up in 30 minutes.

robots.txt and llms.txt are different

If robots.txt tells crawlers "where they may crawl," llms.txt tells them "which content matters." It's a content guide, not access control ^[2].

Who is using llms.txt?

Adoption of llms.txt has grown quickly since it was proposed. Companies with a lot of technical documentation are clearly the early adopters.

What's interesting is that most are companies with extensive technical documentation. The motivation to adopt is much stronger on documentation-centered sites than on blogs or marketing sites. That's also why, for sites centered on marketing content, content structure ranks ahead of llms.txt as a priority.

Measuring impact: what the data says

Opinions on the real impact of llms.txt are still mixed. The bottom line: for now, no direct effect has been clearly proven.

Industry research that analyzed the relationship between having llms.txt and AI citation frequency at scale consistently found that adopting llms.txt showed no statistically significant correlation with AI citations, and pointed to content quality, domain authority, and structured data as the factors that actually affect AI citations. A separate experiment that tracked AI bot crawl logs also found that llms.txt accounted for a very small share of total visits from major AI crawlers like GPTBot and ClaudeBot.

Here's what both results have in common.

1. llms.txt by itself doesn't increase AI citations
2. AI bots tend to collect content through regular web crawling rather than llms.txt
3. What actually affects AI citations is content quality and structure

Should you still set it up?

The data is somewhat negative, but there are reasons it's too early to ignore llms.txt completely.

Reasons to set it up:

1. Nearly zero cost: it takes 30 minutes to make and costs almost nothing to maintain
2. Future-proofing: if AI crawlers start using llms.txt in earnest, sites that already have it will have an edge
3. Documentation cleanup: building llms.txt has the side benefit of organizing your site's core content
4. Industry signal: documentation-heavy companies rushing to adopt it means the technical community is betting on this direction

Where to guard against inflated expectations:

1. Expecting AI citations to rise just by uploading llms.txt doesn't match the data
2. Rather than pouring time into llms.txt, focusing on content structure, schema markup, and adding inline citations is more effective

How to set up llms.txt (6 steps)

The impact isn't guaranteed, but since it costs almost nothing, we recommend setting it up. It doesn't take long to make.

Step 1. Create the file

Create an llms.txt file in UTF-8 in your site's root directory.

Step 2. Write the site information

# TRAIL Labs

> GEO/AEO diagnosis and recommendations for the AI search era. Offers AI visibility diagnosis, citation tracking, and content optimization tools.

Add the site name and a one-line description, so AI can immediately grasp "what this site does."

Step 3. Add key content URLs

## Key content
- [The complete guide to GEO](https://www.traillabs.ai/blog/what-is-geo-optimization): What GEO is and practical strategies
- [SEO/GEO differences in the Korean market](https://www.traillabs.ai/blog/korean-seo-geo-differences): A three-channel strategy for Naver, Google, and AI search

Don't include every page. Select only your top 10–20 pages by traffic and the core content that shows your expertise.

Step 4. Add llms-full.txt (optional)

llms-full.txt is a detailed version that includes the full text of key pages ^[1]. Start with 5–10 pages.

Step 5. Deploy to your server

Confirm it's reachable at https://yourdomain.com/llms.txt.

Also check that the Content-Type: text/plain; charset=utf-8 header is returned.

Step 6. Monitor crawling

Search your server logs for /llms.txt requests. If you see GPTBot, ClaudeBot, and similar in the User-Agent, crawling is happening. As covered above, though, access may be infrequent.

A practical llms.txt template

Here's a template you can use right away.

# [Site name]

> [One-line site description]

## Key information
- [About us](https://yourdomain.com/about): Company mission and team
- [Services](https://yourdomain.com/features): Main features
- [Pricing](https://yourdomain.com/pricing): Price comparison by plan

## Key content
- [Flagship guide](https://yourdomain.com/blog/guide): Comprehensive guide to the core topic
- [Case study](https://yourdomain.com/blog/case-study): Real performance data

## Docs
- [API reference](https://yourdomain.com/docs/api)
- [Getting started](https://yourdomain.com/docs/quickstart)

## Contact
- Email: hello@yourdomain.com

When should it become a priority?

Not every site needs to make llms.txt a top priority. It's realistic to judge priority differently by site type.

For sites other than developer docs, it's reasonable to treat llms.txt as "a low-priority task worth a little time when you have some to spare."

Common mistakes and cautions

The illusion that "uploading llms.txt is enough" is the most dangerous. As the data shows, llms.txt by itself doesn't increase AI citations. Content quality is the prerequisite.

Listing every page blurs what matters. Select only 10–30 key URLs.

Conflicts with robots.txt are another common mistake. Blocking AI crawlers in robots.txt while uploading llms.txt is a contradiction ^[2].

Neglecting updates is also a problem. When your site structure changes, llms.txt needs updating too. We recommend checking it every month.

Splitting time between llms.txt and GEO content work

Comparing where you spend the same time or the same day makes priorities even clearer.

The table shows that rather than spending a day on llms.txt alone, spreading the same time across strengthening FAQs and sources in 10 existing posts is the more sensible choice. We recommend slotting llms.txt in as "something to handle with the time left over after everything else is done."

In the end, llms.txt is a low-priority task: "nice to have, no big loss if you skip it." If you really want to increase AI citations, the realistic order is to first check your content structure and citation signals with a diagnostic tool like TRAIL Search, and leave llms.txt as the next item on the checklist.

More frequently asked questions

When you propose llms.txt to your team, a few questions come up again and again. To "We already have a sitemap, do we need llms.txt too?" you can answer that the roles differ: a sitemap is a list of URLs for search engine crawlers, while llms.txt is a priority guide for AI models. If a sitemap says "here are these URLs," llms.txt says "of these, this is what matters most."

"If we have several language versions, do we need an llms.txt for each language?" also comes up often. For a multilingual site, you can either keep a separate llms.txt per language or split sections by language within one file. But since this is still a proposal with no settled standard, if your site structure is complex, it's also fine to start with just one version in your main language and see how it goes.

To "We already made llms.txt, what's next?" the answer is: don't stop at making it. Periodically check that the links still work and that the list stays current when your top traffic content changes. It's also worth remembering that reinvesting several times that checking time into content structure and source reinforcement produces far more impact than having made llms.txt in the first place.

You can continue with What is GEO (generative engine optimization)? Which pages to start with, in what order, SEO fundamentals in the AI search era: what changed in on-page, off-page, technical SEO, Why llms.txt works as a signpost for AI agents, How to get cited by AI, and AI search optimization tools compared.