If you’ve spent any time in SEO forums or plugin dashboards recently, you’ve probably seen a warning flag up about “llms.txt” – a file some tools now nag you to add. It sounds important. It sounds like something you might be missing. So, should you have one?
Short answer: probably, but not for the reason most people think.
What is llms.txt?
It’s a small text file, written in Markdown, that sits at the root of your website (yoursite.com/llms.txt) – similar to robots.txt or your XML sitemap. The idea, first proposed in 2024, is that it gives AI tools a curated list of your most important pages, so they can find and understand your content more efficiently instead of having to crawl your whole site.
It is not an official web standard. No standards body has ratified it – it’s a community convention that some plugins and platforms have adopted.
Don’t confuse it with blocking AI crawlers
There’s some confusion floating around that’s worth clearing up: llms.txt is not the same thing as controlling which AI bots can access your site.
If you want to block or allow specific AI crawlers (like GPTBot or ClaudeBot), that’s the job of your existing robots.txt file, using standard User-agent / Disallow rules – the same mechanism that’s controlled search engine crawling for decades. That already works today and needs no new file.
llms.txt does the opposite job. It’s not about restricting access – it’s a curated, Markdown-formatted list of your best pages with short descriptions, intended to help AI tools find and understand your content more easily. There’s no “allow” or “disallow” logic in it. If you’re trying to keep AI crawlers off your site, llms.txt won’t help – you need robots.txt (or server-level blocking) for that.
Does it help you rank, or get cited by ChatGPT?
This is the part worth being honest about – because a lot of what’s being sold around llms.txt overstates it.
Google has said directly that llms.txt has no effect on rankings or on what shows up in AI Overviews – positive or negative. Their own John Mueller (Search Advocate and Search Relations team lead) has described it as a minor convenience for AI coding tools reading developer documentation, not something most businesses need to worry about.
It’s a similar story with the AI chat tools themselves. None of the major AI providers – OpenAI, Anthropic, Google, Meta – have confirmed they read or act on llms.txt when generating answers. Independent studies looking for a citation boost from having the file haven’t found one. And in practice, the bots that do occasionally fetch it (GPTBot, ClaudeBot and similar) mostly ignore it in favour of just crawling your normal HTML.
So if the goal is “get my site cited more in ChatGPT or Perplexity” – llms.txt isn’t the lever. What actually drives that is the same thing that’s always driven good SEO: clear, well-structured, genuinely useful content that’s easy for any system (human or machine) to extract information from.
So why would you still add one?
A few reasons it can still be worth doing, just not as a ranking strategy:
- It costs almost nothing. If your CMS or SEO plugin (Yoast, for example) can generate it automatically, there’s no real downside to switching it on.
- It’s genuinely useful for documentation-heavy sites. If you run a product with technical docs that AI coding assistants like Cursor or Copilot need to read, a well-curated llms.txt can help them find the right pages faster.
- It’s a low-cost move. AI-driven search is growing, and llms.txt might become more consequential if major AI providers formally commit to using it. Having one now costs you very little and means you’re not scrambling to catch up later.
- Clients are starting to ask. Even where the SEO case is unproven, it’s increasingly something clients have heard of and want reassurance about.
The risk of doing it badly
If you do add one, the common mistake is treating it like an XML sitemap and dumping every URL on the site into it. That defeats the point. A working llms.txt is short and curated – 20 to 50 genuinely important pages, with plain-language descriptions of what each one covers, not keyword-stuffed SEO copy. It should read like a helpful table of contents, not a ranking attempt.
Also worth avoiding: generating full Markdown copies of every page and making them indexable. That just creates duplicate content, which can actively hurt the pages you care about.
Our verdict
For most of the websites we build and manage – service businesses, councils, clinics, retailers – llms.txt is a nice-to-have, not a priority. It won’t move the needle on traffic or rankings and time is better spent on the fundamentals: solid content, clean technical SEO, and making sure your site is easy for both people and machines to actually read.
Where your CMS can generate it for free, we’ll switch it on as standard housekeeping. Where it needs proper thought – genuinely documentation-heavy sites, or clients who specifically want AI visibility work – we’ll build one properly and treat it as one small piece of a much bigger AI-search strategy, not the whole strategy.
If you’re wondering whether your site needs one, or want a wider look at how AI search might affect your visibility, get in touch and we’ll take a look.