How to Create an llms.txt File: The Complete Automation Guide

Two days ago, PageSpeed Insights flagged my own site under a new "Agentic Browsing" section with one blunt line: file does not appear to contain any links.

My llms.txt was broken. I'd written it once, months ago, and never touched it since.

I fixed it. This post is the "how" — for you, and for future-me the next time I forget how this works.

TL;DR — It's a Markdown file for AI agents, living at yoursite.com/llms.txt. One H1, a short summary, a flat list of links. Don't write it by hand once your site grows — generate it during your build process, then validate before you publish.

What is llms.txt?

It's a plain Markdown file sitting at the root of your domain, written for AI models and agents — not humans, not search crawlers. Think of it as a table of contents for language models: a short summary of what your site is, followed by links to the pages that actually matter.

As more assistants start browsing the web on someone's behalf, giving them a clean map of your content is the easiest way to make sure they find your work and represent it correctly.

How it's different from robots.txt and sitemap.xml

File	Purpose	Audience
robots.txt	Controls crawl access	Search crawlers (Googlebot, Bingbot)
sitemap.xml	Lists URLs for indexing	Search engines
llms.txt	Explains your content	AI agents & LLMs

They solve different problems, so you need all three — not one instead of another.

Does Google actually use it?

Being honest: no search engine currently requires it. It's an emerging convention, not a standard. Some AI systems read it, plenty still ignore it. But it costs almost nothing to maintain, and it's a small, proactive step toward being "AI-ready" as more traffic starts arriving through agents instead of search results.

The structure that works

yoursite.com/
├── robots.txt
├── sitemap.xml
├── rss.xml
└── llms.txt   ← the AI map

Keep the file itself simple:

One H1 — your site name.
A short blockquote summary — who the site is for.
A flat list of links — no deep nesting. Flat scales better inside an AI's context window.

Here's roughly what mine looks like:


## How llms.txt Works  

Unlike `robots.txt`, which tells crawlers what they can or can't access, `llms.txt` doesn't control anything. It simply provides a structured overview of your website in a format that's easy for AI systems to understand.  

When an AI agent visits your website, it may look for `https://yourdomain.com/llms.txt`. If the file exists and the agent supports it, the file acts as a starting point instead of forcing the agent to discover every page on its own.  

A typical workflow looks like this:  

```text  
User asks a question  
        │  
        ▼  
AI agent visits your website  
        │  
        ▼  
Reads /llms.txt (if supported)  
        │  
        ▼  
Understands what the site is about  
        │  
        ▼  
Finds important pages through Markdown links  
        │  
        ▼  
Visits those pages for more detailed information

Instead of crawling hundreds of pages randomly, the agent gets a curated list of your most valuable content. This can make content discovery more efficient for agents that choose to use the file.

Keep in mind that llms.txt is not a replacement for robots.txt or sitemap.xml. It complements them by helping AI agents understand your site's structure and priorities, while traditional search engines continue to rely primarily on existing web standards.

Want to see a real example? Take a look at Xizoa's live llms.txt to see how it's looks on a production website.

Manual vs. automatic

Post count	Method	Verdict
Under 10	Manual	Fine
100+	Automatic	Required

If you're planning to publish anywhere close to 100 pieces of content — which, full disclosure, is exactly the mission I'm on right now — hand-editing this file isn't happening. It's just text output, so treat it the same way you already treat your sitemap or RSS feed: generate it in the build step and forget about it.

Automating it

Python (static site build)

from pathlib import Path

SITE_NAME = "Xizoa"
SITE_URL = "https://xizoa.com"

def generate_llms(posts_registry):
    llm_txt = f"# {SITE_NAME}\n\n> Knowledge base for AI agents.\n\n## Recent Articles\n\n"

    for p in posts_registry:
        llm_txt += f"- [{p['title']}]({SITE_URL}/{p['slug']}/): {p['summary']}\n"

    Path("dist/llms.txt").write_text(llm_txt, encoding="utf-8")

Node.js

const fs = require("fs");
const posts = require("./data/posts.json");

let output = "# Xizoa\n\n> AI-optimized knowledge base.\n\n## Recent Articles\n\n";

posts.forEach(post => {
  output += `- [${post.title}](https://xizoa.com/${post.slug}/): ${post.summary}\n`;
});

fs.writeFileSync("dist/llms.txt", output);

WordPress (functions.php)

function generate_llms_txt() {
    $posts = get_posts(['numberposts' => 50]);
    $output = "# My Site\n\n> Description.\n\n";
    foreach ($posts as $post) {
        $output .= "- [" . $post->post_title . "](" . get_permalink($post->ID) . ")\n";
    }
    file_put_contents(ABSPATH . 'llms.txt', $output);
}
add_action('publish_post', 'generate_llms_txt');

By framework

Hugo — a file at layouts/index.llms.txt, looping through .Pages.
Astro — a llms.txt.js endpoint using getStaticPaths or a simple component.
Eleventy — an llms.njk file in root, with permalink: /llms.txt in the front matter, looping over your collection.
Jekyll — llms.txt in root with empty front matter (--- ---) and a Liquid loop over site.posts.

Validating before you push

[ ] Loads with HTTP 200 and is publicly reachable
[ ] UTF-8 encoded
[ ] Links in proper [Title](URL) Markdown format
[ ] Absolute URLs (https://, not relative paths)
[ ] No duplicate entries
[ ] Opens fine in an incognito browser

Mistakes I'd flag

Missing H1 — agents look for this as the title.
Missing summary — agents need to know what the site even is.
Relative URLs — agents following the link shouldn't have to guess the domain.
HTML instead of Markdown — keep it clean, it's meant to be lightweight.
A heading per article — too much clutter; one "Recent Articles" section is enough.

Quick FAQ

Do I strictly need this? No. But it costs almost nothing and it's a small edge as more discovery happens through agents.

Does ChatGPT read it? When browsing is active, some assistants are starting to honor it — not universally, not yet.

Should every page be listed? No — just the ones that matter. Skip tag pages and other clutter.

This is exactly the kind of small, unglamorous fix that doesn't show up in traffic charts right away but quietly matters later. Fifteen minutes now, and one less broken file the next time something like PageSpeed Insights decides to check.