What Is llms.txt?
llms.txt is a proposed plain Markdown file, placed at yoursite.com/llms.txt, that lists your most useful pages for AI tools to read. It is a community proposal, not a standard. Google says Google Search ignores it, and studies find no link to AI citations, so most sites do not need it.
- llms.txt is a short Markdown list of links with a one-line summary, proposed in September 2024. It is not an official web standard.
- Google says Google Search ignores it. No study we found shows it raises AI citations.
- It costs little to add, so a small test is reasonable, but answer pages matter far more.
- It makes most sense for documentation and developer sites, where tools may fetch pages on demand.
What is llms.txt?
llms.txt is a text file in Markdown format that gives AI tools a short, curated map of your website. Jeremy Howard proposed it in September 2024. The proposal asks sites to add a /llms.txt file with brief background, guidance and links to detailed Markdown pages 1.
It is a community proposal. It is not a W3C or IETF standard, and no search engine owns it 1. The proposal page itself has been revised since 2024, so details may differ from older guides.
Think of it as a reading list written for a machine. Your sitemap lists every page. Your llms.txt says which dozen matter most, and why.
What goes inside an llms.txt file?
The only required element is a heading with your site or project name. Everything else is optional: a short quote-style summary, some plain detail text, and sections of links under second-level headings 1. Each link line is a Markdown list item with a link and, optionally, a note after a colon.
A section called 'Optional' has a special meaning. By convention it holds secondary links that an AI tool can skip when it needs a shorter reading list 1. The proposal also suggests clean Markdown versions of pages, at the same address with .md added 1.
The example below is for a made-up plumbing firm. Swap in your own pages and keep each note to one plain sentence.
- # Example Plumbing Co
- > Licensed residential plumber serving Austin, Texas. We do repairs, water heaters and emergency call-outs.
- ## Services
- - [Water heater replacement](https://example.com/water-heaters): Costs, timing and brands we install
- - [Emergency plumbing](https://example.com/emergency): Hours, response area and call-out fee
- ## Help
- - [How much does a plumber cost?](https://example.com/plumber-cost): Our price ranges, updated yearly
- ## Optional
- - [About us](https://example.com/about): Team, licences and history
Do AI crawlers actually read llms.txt?
There is no confirmed evidence that major AI search engines use it. Google's official guidance says you do not need new machine-readable files or AI text files to appear in Google Search. It adds that doing so will neither harm nor help your visibility, because Google Search ignores them 2. That guidance was last updated on 10 July 2026 2.
Earlier, in April 2025, Google's John Mueller called the file comparable to the keywords meta tag: what a site owner claims about their own site. He also said that, as far as he knew, no AI service had stated that it uses it. He added that server logs suggest the AI services do not even check for the file 3. That was one person's view, but the later official guide agrees for Google.
Log data points the same way. Ahrefs reported that 97% of roughly 38,000 domains with a valid file got zero requests for it in May 2026, from bots or people 4. The sample came from Ahrefs Web Analytics users, so treat it as a rough signal.
Does llms.txt make AI cite you more?
Two independent tests found no link between the file and citations. SE Ranking analysed about 300,000 domains and reported no correlation between having llms.txt and AI citation frequency. It also said removing the variable improved its prediction model 5. The article does not say which AI engines it sampled.
Semrush ran a one-site test. After adding the file to Search Engine Land in March 2025, it found no correlation with better results in AI answers 6. Its logs from mid-August to late October 2025 showed zero visits to the file from GPTBot, PerplexityBot and ClaudeBot 6. That is one site, and no method is given.
Both tests are weak evidence, and neither proves harm. They do show that nobody has demonstrated a benefit.
How many websites have an llms.txt file?
It depends on who counts, and the answers differ widely because the samples differ. Rankability checked the top 10,000 domains in September 2026 and confirmed 830 (8.3%) serving a valid file 7. SE Ranking found 10.13% of about 300,000 domains 5. Ahrefs found 28% in a sample of 137,000 sites that use its analytics, which skews to tech-aware owners 4.
Common Crawl offers a reality check. In its July 2026 crawl, 11.72% of sampled /llms.txt URLs returned a text body, and it warns that the sample is not representative of the web 8. So adoption is real but uneven, and these samples cannot tell you how it varies by site type.
Adoption is also not the same as use. A file that exists and a file that a machine reads are two different facts.
| Source | Sample | Share with llms.txt |
|---|---|---|
| Rankability, Sept 2026 7 | Top 10,000 domains | 8.3% (830 sites) |
| SE Ranking, Oct 2026 5 | About 300,000 domains | 10.13% |
| Ahrefs, May 2026 4 | 137,000 domains using Ahrefs Web Analytics | 28%, likely an upper bound |
| Common Crawl, July 2026 8 | Sampled /llms.txt URLs | 11.72% returned a text body |
Do you need an llms.txt file?
Most small businesses do not need one, and none should buy it as a growth service. The work that moves AI visibility is elsewhere: crawlable pages, direct answers and consistent mentions of your name. Google's guide tells site owners to prioritise effective SEO over AEO and GEO hacks, and names llms.txt as an unnecessary tactic for Google Search 2.
Some sites have a better case. Documentation, API and software sites are where people point coding assistants at a URL and ask questions. If your help centre already works that way, a curated file is a cheap courtesy. Anthropic's own developer documentation publishes one with hundreds of links to Markdown pages 9. Some documentation platforms, such as GitBook and Mintlify, publish the file for their customers (check your platform's docs).
| Your situation | Add llms.txt? | Why |
|---|---|---|
| Local service business | Skip for now | Buyers use search and AI answers; Google Search ignores it 2, and no study shows AI citations improve |
| Online shop or lead-gen site | Optional test | Cheap, but fix page quality first |
| Software docs or API reference | Reasonable | Tools may fetch docs on demand; check whether your docs platform already publishes one |
| Site with thin or blocked pages | No | Fix crawl access and content before adding a map to it |
How is llms.txt different from robots.txt and sitemap.xml?
Each file does a different job. robots.txt tells bots what they may crawl. A sitemap lists pages you want indexed. llms.txt is meant to be read on demand by an AI tool that needs a quick overview of a site 1.
The proposal says a sitemap is too large to fit a context window and leaves out machine-readable versions of pages 1. For the full side-by-side, read llms.txt vs robots.txt vs sitemap.xml. If your real worry is which AI bots may visit, the guide on allowing AI crawlers in robots.txt matters more.
What did Claude and Gemini search when asked about llms.txt?
On 2026-10-20 we asked Gemini and Claude 'what is llms.txt and do I need it' and recorded their web searches. Claude ran three searches, including one on adoption statistics and one on whether crawlers read the file. Gemini ran two short searches. Sources they drew on included documentation platforms such as GitBook and Mintlify, plus vendor blogs.
This suggests readers want proof of use, so this page leads with sourced numbers.
| Engine | Search it ran |
|---|---|
| Gemini | llms.txt |
| Gemini | what is llms.txt |
| Claude | what is llms.txt file |
| Claude | llms.txt adoption statistics 2026 |
| Claude | do AI crawlers read llms.txt |
How can you apply this week?
Spend your first hour on the basics, and treat llms.txt as the last item.
- Check that search bots can reach your site with the free AI crawler checker.
- Write answer-first pages for your five top buyer questions. Our guide on writing content AI engines quote shows how.
- If you still want the file, build it with the llms.txt generator and follow how to create an llms.txt file.
- Check your server logs after 30 days. If no bot ever requests the file, you have your answer for your own site.
What an AI assistant searches when it answers this
We gave the question below to AI search tools and recorded the web searches they ran behind the scenes. This is called query fan-out. Each phrase is a page you could publish to be found and cited.
“what is llms.txt and do I need it”
The searches it ran- what is llms.txt file
- llms.txt adoption statistics 2026
- do AI crawlers read llms.txt
- llms.txt
- what is llms.txt
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal made by Jeremy Howard in September 2024 1. No standards body or search engine has adopted it, and the proposal page has been revised since. Treat any guide that calls it a standard with suspicion.
Where do I put the llms.txt file?
At the root of your site, so it loads at yoursite.com/llms.txt. The proposal also allows a subpath 1. Serve it as plain text or Markdown with a normal 200 status, and make sure your robots.txt does not block it.
Will llms.txt hurt my SEO?
Google says creating the file will neither harm nor help your visibility in Google Search 2, and it describes making one for other services as completely fine. The real cost is your time and the risk of stale links, so keep it short and review it each quarter.
What is the difference between llms.txt and llms-full.txt?
llms-full.txt is a larger variant that some sites publish with full page text in one file. The original proposal page we read does not describe it. Originality.ai counted 2,463 sites with one in May 2026, versus 36,120 with llms.txt 10, so it is much rarer.
Should I pay someone to create llms.txt?
No. It takes under an hour with a generator, and the evidence for any benefit is weak 5. Spend money on answer pages and crawl access instead. Be wary of any provider that sells the file as a way to win AI citations.
Sources
- The /llms.txt file · llmstxt.org (Jeremy Howard)Opened October 20, 2026. Supports: The proposal (September 2024, later revised): Markdown file at /llms.txt, H1 only required element, optional summary, H2 link lists, Optional section, .md page versions; community proposal rather than a formal standard; comparison with sitemap and robots.txt.
- Optimizing your website for generative AI features on Google Search · Google Search CentralOpened October 20, 2026. Supports: Google Search ignores llms.txt, which neither helps nor harms visibility; no new machine-readable files needed; fine to create for other services; prioritise effective SEO over AEO/GEO hacks (last updated 2026-07-10).
- Google's John Mueller: llms.txt comparable to the keywords meta tag · Search Engine JournalOpened October 20, 2026. Supports: In April 2025 Mueller said llms.txt is comparable to the keywords meta tag and that none of the AI services had said they use it (secondary report of an individual's comment, not formal policy).
- What is llms.txt? · AhrefsOpened October 20, 2026. Supports: 28% of 137,000 Ahrefs Web Analytics domains publish llms.txt (skewed sample); 97% of about 38,000 valid files received zero requests in May 2026 (page updated 15 June 2026).
- LLMs.txt study · SE RankingOpened October 20, 2026. Supports: About 300,000 domains; 10.13% had llms.txt; no correlation with AI citation frequency; removing the variable improved model accuracy (published 15 October 2026; engines and period not disclosed).
- What is llms.txt? · SemrushOpened October 20, 2026. Supports: One-site test on Search Engine Land from March 2025: no correlation with improved AI results; logs mid-August to late October 2025 showed no visits from GPTBot, PerplexityBot or ClaudeBot and only a few from Googlebot and Bingbot.
- LLMs.txt adoption data · RankabilityOpened October 20, 2026. Supports: Scan on 18 September 2026 of the top 10,000 Tranco domains: 830 (8.3%) serve a valid llms.txt; blocked or timed-out sites stay in the denominator.
- A content analysis of llms.txt files from the July 2026 crawl archive · Common CrawlOpened October 20, 2026. Supports: For /llms.txt, 11.72% of sampled URLs returned a text body in the July 2026 crawl; the sample is not representative of the whole web (published 31 August 2026).
- Anthropic Developer Documentation llms.txt · Anthropic (platform.claude.com)Opened October 20, 2026. Supports: Anthropic's developer documentation publishes an llms.txt file listing hundreds of links to Markdown pages (read on 2026-10-20).
- llms.txt tracking study · Originality.aiOpened October 20, 2026. Supports: In May 2026 its tracker counted 36,120 sites with llms.txt and 2,463 with llms-full.txt, within a subset of the web (article dated 3 July 2026).