What is llms.txt supposed to do?
The proposal (from Answer.AI's Jeremy Howard, 2024) is reasonable: LLMs have small context windows and websites are noisy, so give models a clean markdown index — site purpose, key pages, canonical descriptions — at /llms.txt, with optional /llms-full.txt containing expanded content.
Think of it as a sitemap for reasoning engines rather than crawlers. The idea's quality was never the issue; adoption by the engines is.
What's the actual adoption status?
As of mid-2026: OpenAI, Anthropic, Google and Perplexity have not confirmed production consumption of llms.txt. Google's search team has publicly called it unnecessary for AI features, comparing it to the failed keywords meta tag. Crawl-log studies show AI bots occasionally fetching the file without evidence it changes retrieval or citation behavior.
Meanwhile the GEO factors with measurable effect — answer structure, schema, third-party presence, original data — are documented across multiple studies. The opportunity cost of confusing the two is the real risk.
- No major provider confirms production use
- Google: explicitly unnecessary
- Occasional bot fetches ≠ retrieval influence
- Measured GEO levers live elsewhere
So why does Chalk Labs ship one anyway?
Because it costs five minutes, has zero downside, and functions as a tidy canonical-description artifact even if only humans and the odd crawler read it. Ours is generated automatically from the same data that builds this site's pages — you can read it at /llms.txt.
The rule we give clients: ship llms.txt the way you'd ship a favicon — basic hygiene, no strategy weight. If a vendor's 'AI optimization' deliverable leads with llms.txt, you've learned something important about the vendor.
Questions we hear about this
No. It's inert at worst. The only harm is opportunity cost — treating it as the strategy while skipping answer structure, schema, entity work and third-party presence, which is where measured GEO effects actually live.
A one-paragraph site description, then grouped markdown links to your most important pages with one-line descriptions each. Keep it under a few hundred lines, current, and consistent with how you describe yourself everywhere else — consistency is the only part that plausibly matters.
Only if it's automated from your content pipeline. Hand-maintaining a parallel full-text corpus for speculative consumption is effort better spent refreshing the actual pages engines demonstrably read.
The measured stack: allow AI crawlers, server-render, answer-first formatting, FAQPage/Organization schema, named authorship, original statistics, and consistent third-party brand presence. Boring, documented, effective — the opposite of a magic file.