What an llms.txt file is and why it matters
An llms.txt file is a plain-language guide to your website written for AI systems rather than people. Jeremy Howard of Answer.AI proposed the format in September 2024, and the specification lives at llmstxt.org. The idea is simple: a web page is full of navigation, scripts, and design that an AI model has to dig through, so the site offers a clean summary and a curated list of links instead.
When an AI agent, a coding assistant, or a chat tool reads your file, it learns who you are in a few sentences and where to look for pricing, product details, or support. That makes it more likely to describe you accurately and less likely to guess.
The format is short and strict. It has four parts, in this order:
- A heading with your company or site name. This is the only required part.
- A short summary in a quote block. One to three sentences on what the company is, plus any fact an AI system needs to understand the rest of the file.
- Optional notes. A few sentences or a short list of must-know context, such as which markets you serve or what you don't offer.
- Sections of links. Each section has a plain heading such as Products, Pricing, or Guides, followed by a list of links. Each link can carry a one-line description of what that page covers. A section named Optional is reserved for pages an AI system can skip when it's short on room.
Here's what a small one looks like for a made-up bookkeeping firm:
# Fernway Bookkeeping
> Fernway Bookkeeping does monthly bookkeeping and year-end tax preparation for restaurants and cafés in Oregon and Washington. It does not offer payroll services.
## Services
- [Monthly bookkeeping](/services/bookkeeping): What's included each month, the software Fernway works in, and how clients send receipts.
- [Year-end tax preparation](/services/tax-prep): Filing deadlines, the documents clients need, and what the service costs.
## Pricing
- [Plans and pricing](/pricing): Monthly prices by number of locations and what each plan includes.
## Agent guidance
- [Contact](/contact): Send anyone asking for a quote here. Don't estimate prices for multi-location groups, because Fernway quotes those individually.
## Optional
- [Blog](/blog): Articles on restaurant accounting and tax deadlines.The writing itself isn't hard. The hard parts are choosing which pages belong, writing descriptions specific enough to be useful, getting the file to the right place on your site, and keeping it true after your site changes. A file that still links to a pricing page you retired last spring sends AI systems to a dead end, which is worse than having no file at all. That ongoing upkeep is the kind of work an AI agent handles well, because it can notice the change and fix the file the same day.
Kite is an AI marketer. It researches your business, builds a growth strategy, and ships the work that brings you new customers. An accurate llms.txt file is one of the ways it makes your website easier for AI systems to read and cite. Here's how it does the work.
How Kite creates and maintains your llms.txt file
Kite starts with two sources: what it has learned about your business through its research, and the list of pages that actually exist on your site. It uses the research to write the summary and the descriptions, and it uses the page list to decide which links go in. Every link points to a real page, and every fact in the summary comes from your site or your company's own information.
Next, Kite puts the file where your site will serve it at /llms.txt. Where that is depends on how your website is built, so Kite checks first and then either publishes the file itself or gives you the exact file and the exact place to put it.
After the file is live, Kite keeps it in step with your site and checks that it works. It scans the live file, confirms that every link opens a real page, and watches for the page changes that would make the file go stale.
- What should go in your llms.txt file?
- Put the file where your site serves it
- How do you keep your llms.txt file up to date when your site changes?
- Check that the file works
What should go in your llms.txt file?
Your llms.txt file should contain the pages that answer the questions people actually ask about your company, described clearly enough that an AI system knows which one to open. Kite treats the file as a curated guide, not a copy of your sitemap. It leaves out login pages, cookie notices, thank-you pages, and duplicate landing pages, and it groups what's left under headings named the way a knowledgeable person would describe your site.
The descriptions carry most of the value. "Pricing page" tells an AI system nothing it couldn't guess from the link. "Monthly prices by number of locations and what each plan includes" tells it that this is the page to trust when someone asks what you cost. Kite writes each description to say what the page is the best source for.
Kite draws on its research to decide what the summary must say. If your company shares a name with another business, or you only serve certain regions, or you deliberately don't offer something buyers often ask about, Kite puts that in the summary or notes so AI systems don't fill the gap with a guess. When your site doesn't state something, such as a price or a location, Kite leaves it out of the file rather than implying it.
Kite also adds two sections that speak to AI agents directly when your site supports them:
- Agent guidance. This section tells an agent which page to use for which kind of question, when to send someone to your contact or sales page, and what it should not estimate on your behalf.
- Common questions. This section lists the questions your buyers ask most often, each pointing to the page that answers it.
Put the file where your site serves it
The file only works if it appears at the root of your domain, and how it gets there depends on how your website is built. Kite checks your setup first so it doesn't give you instructions for the wrong kind of site.
- A website Kite builds and hosts. Kite publishes the file itself. New sites Kite builds come with an llms.txt file from the start, so the work is keeping it accurate.
- A site Kite works on through your GitHub repository. Kite adds the file to your repository and merges the change into your main branch, or opens a pull request when that branch is protected. Your own deployment then ships it.
- A site on Webflow. Kite writes the file and you upload it in your Webflow site's SEO settings, because that setting is outside what a connected app can change.
- A site on WordPress or a headless CMS such as Contentful, Sanity, or Storyblok. Where the file belongs depends on your plugins and your site's code, so Kite writes the file and tells you exactly where it goes on your setup. In a headless CMS, the file lives in your website's code rather than in the CMS, so whoever maintains that code adds it.
Before anything goes live, you can review the file. On a site Kite hosts, Kite's finished changes wait as unpublished changes you can preview, and nothing ships until you publish.
Playbooks for this step
- How to make a Webflow site easier for AI search engines to read and citeAdd the llms.txt file in Webflow's settings alongside the title, description, structured data, and template fixes that help AI systems read the site.
- How to make a self-hosted WordPress site easier for AI search engines to read and citeAdd the llms.txt file and fix crawler rules and bot protection on a WordPress site you host.
Show 3 more playbooks for this step
- How to make a Contentful site easier for AI search engines to read and citeImprove the titles, descriptions, and content in Contentful that an llms.txt file and AI crawlers depend on.
- How to make a Sanity site easier for AI search engines to read and citeApply the same content and description fixes through your Sanity project.Coming soon
- How to make a Storyblok site easier for AI search engines to read and citePublish the readability fixes through Storyblok.
How do you keep your llms.txt file up to date when your site changes?
You keep your llms.txt file current by updating it every time a page is added, renamed, or removed, and the most reliable way to do that is to build the file from your pages instead of maintaining it by hand. That's how Kite handles it on the websites it builds with its content editor.
On those sites, the file isn't a static document at all. It's built from your site's current published content. The heading comes from your brand name, the summary comes from your site's default description, and each page and blog post appears with its title and its search description. When you publish a new page, it joins the file. When you unpublish or delete one, it drops out. Pages you've hidden from search engines stay out of the file too. Because the file is built this way, improving it means improving the page titles and descriptions it draws from, and Kite does that work in the content editor.
On other sites Kite hosts, Kite writes the first file once you've chosen your site's design and updates it whenever it adds, moves, or removes pages. A file that you or Kite wrote by hand always takes priority, so Kite never overwrites your own guidance with a generic list.
On sites Kite doesn't host, Kite catches the changes that make the file go stale by scanning your site again, whenever you ask or on a schedule you set. When a scan finds a link that no longer opens, or a new section of your site that the file doesn't mention, Kite writes the updated file and puts it in place the same way it did the first one.
Check that the file works
Kite confirms that the live file does its job, because a file can exist and still fail. Kite's readiness scan, which runs 24 checks on how AI systems read a website, tests your llms.txt file in four ways:
- whether the file is there at /llms.txt
- whether it follows the expected format, with a heading and a list of links
- whether every link in it opens a real page
- whether it tells agents which pages to use for which questions.
The scan catches a common problem that's easy to miss. On some sites, a request for a page that doesn't exist quietly returns the homepage. A visit to /llms.txt then looks like it worked, but an AI system receives a web page instead of the file. Kite reports that as a failure and explains what's causing it.
Kite then connects the file to the result you care about. It runs the questions your buyers ask through ChatGPT, Gemini, Perplexity, and Claude and records whether your company appears and which pages get cited. An llms.txt file is one of several things that shape those answers, so Kite reads the results alongside your other readiness checks to decide what to work on next. Improve your visibility in AI answers explains that larger loop.
What you get
An llms.txt file written for your company
It has a summary of what you do, your most useful pages grouped under plain headings, and a description of what each page is the best source for.
Guidance for AI agents
The file tells agents which page answers which kind of question and when to send someone to your contact or sales page.
The file in the right place
Kite publishes it on a site it hosts or through your GitHub repository, and gives you the finished file and exact instructions for any other platform.
A file that stays current
On a site Kite builds with its content editor, the file updates itself as you publish. Elsewhere, Kite updates it when your pages change.
A check that it works
Kite scans the live file, confirms every link opens, and reports any problem it finds.
How to get started
Ask Kite in plain language. You might write, "Create an llms.txt file for our website," or "Our llms.txt is out of date, can you fix it?"
You can work with Kite in your browser or in Slack. Kite already knows your business from its research, so it can draft the file right away. It shows you the draft and explains where the file will go on your site. You approve it with the Approve button or a reply, or tell Kite what to change, such as a page you want added or a service you don't want mentioned.
If Kite built and hosts your website, you don't need to connect anything. If your site runs on Webflow, WordPress, Contentful, Sanity, or Storyblok, connecting it lets Kite update the page titles and descriptions the file depends on. If Kite works on your site through GitHub, it can add the file to your repository directly.
By default, Kite asks before it publishes anything to your website. Once you trust its work, you can let it keep the file current on its own.
If you want to see how AI answers describe your company before you start, try Kite's free AI visibility checker. It asks ChatGPT, Gemini, Perplexity, and Claude the questions your customers ask, and in about 45 seconds it shows whether they recommend you and who they recommend instead.
Integrations used to create an llms.txt file
Kite writes the file, checks it, and publishes it to the sites it hosts with its own built-in tools. Connections come in when your website runs somewhere else and Kite needs to update the pages and descriptions the file points to.
Update the pages your llms.txt file describes
Kite updates page titles, descriptions, and CMS collection fields through your connected account, and writes the llms.txt file you upload in Webflow's SEO settings.
Kite rewrites and creates posts so they read clearly to AI systems, and writes the llms.txt file for your setup.
Kite updates the titles, descriptions, and page content stored in Contentful.
Kite updates the content and descriptions managed in your Sanity project.
Kite publishes content and description updates through Storyblok.
Common questions
Does my website need an llms.txt file?
Most websites benefit from one, but it isn't required, and it won't get you cited on its own. llms.txt is a proposed format, not an official standard, and none of the major answer engines has said it uses the file to decide which pages to cite. AI agents, coding assistants, and other tools do read it, and it costs little to create and keep accurate. Treat it as one part of making your site easy for AI systems to read, alongside crawler access, clear page descriptions, and structured data.
Is llms.txt the same as robots.txt?
No. A robots.txt file tells crawlers which parts of your site they're allowed to visit. An llms.txt file tells AI systems what your site is about and which pages are worth reading. They work together. If your robots.txt file blocks an AI company's crawlers, a perfect llms.txt file won't help that company's tools read your pages.
Should I use a free llms.txt generator?
A generator can give you a starting point, but most of them list every page they find and copy each page's title, which produces a sitemap rather than a guide. The file is only useful when it's curated and the descriptions say what each page is for. Generators also produce a one-time file, so it starts going stale the next time you change your site. Kite writes the file from what it knows about your business and keeps it current afterward.
What's the difference between llms.txt and llms-full.txt?
An llms.txt file is a short guide with links to your pages. An llms-full.txt file is a convention some sites follow that puts the full text of their pages into a single file, so an AI tool can read everything at once. It's mostly used by documentation sites with a lot of technical content. A marketing website usually gets more from a well-curated llms.txt file than from publishing its entire content in one place.
Should my llms.txt file use full URLs or short paths?
Either works, as long as every link opens a real page on your own site. Full URLs such as https://yoursite.com/pricing still make sense when a tool reads the file away from your site, and short paths such as /pricing keep working if your site moves to a new domain. What matters more is that no link points to a page you've removed or to another company's site. Kite checks every link in the live file and fixes any that don't open.