Does llms.txt work? What crawler data shows

An llms.txt file is a Markdown file at a website's root that lists the site's most important pages for AI tools, and as of October 2026 neither Google, OpenAI, Anthropic, nor Perplexity states that its crawlers read site owners' llms.txt files. Google states that Google Search ignores llms.txt and that the file neither helps nor harms search visibility. On ericgrows.com, which has published an llms.txt since July 2026, Cloudflare analytics recorded zero requests for the file from verified AI crawlers between September 10 and October 10, 2026, while ClaudeBot, OAI-SearchBot, and Googlebot requested robots.txt between 152 and 312 times each. An llms.txt file is harmless and quick to add, and no evidence found links it to being cited by AI.

Updated: October 10, 2026. Written by Eric, founder of Eric Grows Companies.

What is an llms.txt file?

An llms.txt file is a plain Markdown file placed at a website's root, at /llms.txt, containing a title, a short summary of the site, and lists of links to the site's most important pages. Jeremy Howard proposed the llms.txt format on September 3, 2024, at llmstxt.org, and the proposal reached version 2 in August 2026. The proposal describes the file as information to help AI agents use a website on demand while assisting a user, and notes that it is used most heavily for software documentation, where coding agents follow it.

Can llms.txt block AI from training on a website?

No. An llms.txt file contains no allow or disallow rules, so it cannot block or permit any crawler, and the llms.txt proposal itself identifies robots.txt as the file for access control. A website owner who wants to control which AI crawlers may use the site does so in robots.txt, which OpenAI, Anthropic, and Perplexity each document as the file their crawlers follow. Pages claiming that GPTBot or other crawlers obey rules in llms.txt are not supported by the AI companies' own documentation.

Do AI crawlers actually request llms.txt?

On ericgrows.com, verified AI crawlers did not request llms.txt at all between September 10 and October 10, 2026, according to Cloudflare analytics counting only bots Cloudflare verifies. In the same 30 days, ClaudeBot requested robots.txt 312 times and sitemap.xml 314 times, OAI-SearchBot requested robots.txt 189 times, GPTBot requested sitemap.xml 61 times, and Googlebot requested robots.txt 152 times. The roughly 27 requests the llms.txt file did receive came from SEO tools, website scanners, ordinary browsers, command-line fetches, one verified data-provider crawler, and two unverified requests claiming to be GPTBot. The data covers one site for 30 days, Cloudflare counts are sampled, and Perplexity's crawlers made no verified visits in the window.

Requests by Cloudflare-verified crawlers on ericgrows.com, September 10 to October 10, 2026
Crawlerllms.txtrobots.txtsitemap.xml
ClaudeBot (Anthropic)0312314
OAI-SearchBot (OpenAI, ChatGPT search)01890
GPTBot (OpenAI)0061
ChatGPT-User (OpenAI)000
Googlebot015213
Applebot091
Amazonbot000

Does Google use llms.txt?

No. Google's guidance for site owners on AI optimization states that Google Search does not use llms.txt files, that site owners may create them for other services, and that doing so will neither harm nor help a site's visibility or rankings in Google Search. Google's separate documentation on AI features states that appearing in AI Overviews and AI Mode requires no new machine-readable files or AI text files. Google's Lighthouse tool includes llms.txt in its agentic browsing audit as an emerging convention and treats a missing file as not applicable rather than as a failure.

Do OpenAI, Anthropic, and Perplexity read llms.txt?

OpenAI, Anthropic, and Perplexity do not state that their crawlers read site owners' llms.txt files. OpenAI's documentation for OAI-SearchBot, GPTBot, and ChatGPT-User, Anthropic's documentation for ClaudeBot, Claude-User, and Claude-SearchBot, and Perplexity's documentation for its crawlers all describe robots.txt as the controlling file. Each of the three companies publishes an llms.txt for its own developer documentation, such as developers.openai.com/llms.txt and docs.perplexity.ai/llms.txt. Publishing an llms.txt for one's own documentation is not evidence that a company's crawlers read other sites' files.

Is there evidence that llms.txt improves AI citations?

No published evidence links llms.txt to AI citations. A third-party study by SE Ranking, an SEO software company, of about 300,000 domains found that roughly 10% had an llms.txt file and that having one showed no correlation with how often a site was cited by AI. Google's statement covers Google Search only, and the ericgrows.com data measures crawler requests rather than citations, so the available evidence shows no effect on AI citations without proving that none exists.

When is an llms.txt file worth having?

An llms.txt file is worth having when it costs nothing or serves a specific audience. Several website platforms and SEO plugins, including Yoast, AIOSEO, and Wix, can generate the file automatically, so keeping it is free. Software documentation sites benefit most, because the llms.txt proposal notes that coding agents follow llms.txt files in software documentation. For a typical small business website, an llms.txt is optional and is not a reason to pay an agency.

What should a business do instead of relying on llms.txt?

A business that wants to be cited by AI should first make sure AI search crawlers can reach its website, because a block can sit at the hosting company or security service rather than in robots.txt. In audits of 23 self-selected business websites by the Eric Grows Companies free AI visibility assessment between August 18 and October 4, 2026, 22% had a ChatGPT, Claude, or Perplexity crawler blocked at that level. The next steps are stating key business facts on the page as readable text and earning mentions on other credible websites, as covered at https://ericgrows.com/learn/get-chatgpt-to-recommend-your-business. The free assessment at https://ericgrows.com/audit checks AI crawler access.

Does ericgrows.com have an llms.txt file?

Yes. ericgrows.com publishes an llms.txt at https://ericgrows.com/llms.txt listing the site's guides and main pages, maintained by hand and updated whenever a guide is published, and served both to visitors and on the site's separate AI-readable surface. The file costs little to maintain, and the site relies on crawler access, readable pages, and outside mentions for its AI visibility rather than on llms.txt. The site's own crawler data, above, is the basis for this guide's conclusion.

Who wrote this guide?

This guide is written by Eric, founder of Eric Grows Companies. Eric was a Senior Manager of Strategy and Operations at Meta, holds an MBA and a CPA, and has never written a line of code. Eric runs three live companies himself with AI, StratEngine AI, SCUBAzen, and Venice Threads Co., built the ericgrows.com AI visibility assessment, and teaches non-technical business owners to get found by AI answer engines.

What are the sources for this guide?

Where can I read more?

Each guide takes one part of running a business with AI end to end. The hub guide below covers the whole path.


This is the AI-readable version of this guide. The primary page for people is at https://ericgrows.com/learn/does-llms-txt-work.