llms.txt and AI Crawlers: Should You Block or Welcome Them?

Let me start with the part you probably came for.

llms.txt does not help your SEO, and the data on this is now fairly damning. Google's June 2026 documentation states plainly that Search doesn't use it — not for rankings, not for AI Overviews, not at all. SE Ranking studied 300,000 domains and found about 10% adoption; when they modelled whether the file correlated with AI citation frequency, removing the llms.txt variable actually improved their model's accuracy. It was adding noise, not signal.

Limy.ai monitored over 500 million AI bot events across 90 days and found only a few hundred requests targeting /llms.txt at all. Gary Illyes has said Google doesn't support it and isn't planning to. John Mueller compared it to the keywords meta tag — which is a genuinely brutal comparison, and a fair one, since both are files where you tell search engines what your site is about with no verification whatsoever.

So that's settled. But there's a more interesting question underneath it, and it's the one worth your time: which AI crawlers should reach your site at all?

The short version

llms.txt: not an SEO tactic. Possibly worth twenty minutes if you have documentation that coding agents or agentic browsers might read. Not otherwise.

Crawler access: genuinely important, and the key distinction is that training crawlers and retrieval crawlers are now separate bots.

Purpose	Bots	If you block them
Training	GPTBot, ClaudeBot, Google-Extended, CCBot	Content isn't used for model training. No effect on Google rankings.
Retrieval	OAI-SearchBot, Claude-SearchBot, PerplexityBot	You disappear from AI answers and citations.

Blocking the first group is a legitimate choice. Blocking the second group by accident is one of the most common and most costly AI visibility errors there is.

Why llms.txt exists

The idea is reasonable enough: a Markdown file at your domain root giving AI systems a curated map of your most important content, so they don't have to infer structure from HTML.

The problem is that it's a community convention, not a standard. No W3C or IETF backing, no enforcement mechanism, no commitment from any major provider to read or act on it in production. Anthropic, OpenAI, and Google all publish llms.txt for their own documentation while directing site owners to robots.txt for actual crawler management — which tells you how they think about it.

Mueller's criticism cuts to the real weakness: a file where you declare what your site is about, with nothing verifying the claim, is structurally gameable. That's exactly the story of the keywords meta tag, and it ended the same way both times.

The case that isn't SEO

Here's the nuance, because "it's useless" isn't quite right either.

Stripe, Vercel, Cloudflare, Anthropic, and Mastercard all ship the file. They aren't confused about the search data. They're building for a different audience: coding agents and agentic browsers that navigate sites to complete tasks or retrieve documentation.

There's a visible split inside Google on exactly this point. Search Central says skip it. Meanwhile Chrome's Lighthouse moved llms.txt auditing out of experimental status and into a default "Agentic Browsing" category in May 2026. Both teams are right on their own terms, because they're optimising for different things.

So the honest position: if you publish technical documentation that agents might traverse, llms.txt is twenty minutes well spent. If you run a marketing site or a blog and you're doing it for search visibility, you're maintaining a file for no return — and the llms-full.txt variant, which can run to hundreds of thousands of words on a large site, is a genuine ongoing maintenance cost for nothing.

The crawler decision that actually matters

Now the part with real consequences.

Training crawlers

GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Gemini training), CCBot (Common Crawl), Applebot-Extended.

Blocking these prevents your content being used to train models. Two things worth knowing:

Blocking GPTBot or Google-Extended has no effect on your Google Search rankings. Googlebot and Google-Extended are separate, and your rankings are determined by the former.

Blocking training also doesn't remove you from AI answers, because answers increasingly come from live retrieval rather than training memory.

Whether to block is a values question — about consent and compensation for your work — rather than a marketing one. Reasonable people land differently, and I don't think there's a single right answer here.

Retrieval crawlers

OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User.

These fetch content to answer questions with citations. They're the mechanism by which you appear in AI answers at all.

OpenAI's documentation is unambiguous: sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers — regardless of whether GPTBot crawled your content previously. The two systems are independent.

This is where people get hurt. Someone reads "block AI crawlers," pastes in a list that includes the retrieval bots, and quietly removes themselves from ChatGPT. Nothing in their analytics tells them.

A working robots.txt

Block training, stay eligible for citation:

# Traditional search — always allow
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI retrieval — allow, these enable citations
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# AI training — block if that's your position
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Persistently non-compliant
User-agent: Bytespider
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Want maximum visibility and don't object to training? Allow everything except Bytespider.

Two things robots.txt won't do for you

Your CDN may be overriding it. Cloudflare and similar offer one-click AI bot blocking, and it operates at a layer your robots.txt never sees. Plenty of sites have a perfectly permissive robots.txt and a CDN quietly returning 403s to OAI-SearchBot. Check your server logs for 403 responses to the retrieval bots — that's the only way to know what's really happening.

Some bots ignore it entirely. Bytespider has been widely documented disregarding disallow rules, and Cloudflare published evidence in August 2025 of Perplexity using undeclared crawlers that rotate user agents and IPs to evade directives. Against non-compliant bots, robots.txt is a request rather than a control — the only real enforcement is at the server or WAF level.

There's also a category robots.txt was never designed for: live user-triggered fetches. When someone asks an assistant to read a specific page, that fetch is arguably not crawling at all, and may not respect robots.txt regardless.

Frequently asked questions

What is llms.txt?
A plain-text or Markdown file at your domain root offering AI systems a curated map of your most important content. It's a community convention rather than a formal standard — no W3C or IETF backing, no enforcement, and no major AI provider has committed to using it as a citation signal in production. Google's June 2026 documentation states Search doesn't use it at all.

Should I block GPTBot?
That's a values decision rather than a marketing one, and it carries less cost than people assume. Blocking GPTBot prevents your content being used for OpenAI's model training and has no effect on your Google rankings or your eligibility for AI citations. What you must not do is block OAI-SearchBot by mistake — that's the retrieval crawler, and blocking it removes you from ChatGPT's search answers entirely.

Does blocking AI crawlers hurt traffic?
Blocking training crawlers doesn't measurably affect search traffic or AI citation eligibility. Blocking retrieval crawlers does — you become ineligible to appear in AI answers, and AI referrals convert at notably higher rates than organic in several analyses. Also check your CDN, since AI-blocking features there can block retrieval bots without any change to your robots.txt.

What to do next

Open yourdomain.com/robots.txt and look for OAI-SearchBot, Claude-SearchBot, and PerplexityBot. If any are disallowed and you didn't decide that deliberately, fix it today.

Then check your server logs for 403 responses to those same user agents. A permissive robots.txt and a blocking CDN is a common and completely invisible combination.

As for llms.txt — unless you publish documentation for agents, skip it and spend the time on your opening paragraphs instead.

Free: The SEO audit checklist.

---

Related guides

• SEO when AI answers the question first — the wider strategy
• How to track AI search traffic — measuring what the bots produce
• robots.txt mistakes — the errors that de-index whole sites