Google Just Pulled the Last Prop Under llms.txt: 97% of Files Across 137K Sites Are Never Read, AI Retrieval Bots Are 1.1%

On the evening of 17 September 2026 UTC, John Mueller of Google Search Relations posted one observation on Bluesky that cut a year of llms.txt "proof" in half. He noted that a crop of "everything llms.txt" directory sites now exists: they scan every domain they can find and link to whatever files they discover, so any crawler visiting those directories follows the links. Translated: that llms.txt fetch in your log probably isn't an AI platform seeking you out — it's a bot picking your file up off somebody else's list.

llms.txt is a single index-file specification proposed in 2024 by Jeremy Howard, co-founder of Answer.AI: Markdown at your site root telling models who you are and which pages matter. The SEO industry spent eighteen months turning it into a product — Wix generates it, Framer and Lovable scan for it, WordPress plugins emit it in one click, and GEO/AEO tools wrote "has llms.txt" into their scorecards. Adoption and evidence never lined up. SE Ranking scanned nearly 300,000 domains: 10.13% had the file, and the highest-traffic tier adopted it least at 8.27%. Worse, when they built an XGBoost model to predict how often a domain gets cited by LLMs and removed llms.txt as a variable, the model got more accurate. The field wasn't signal. It was noise.

Compare three standards that worked. robots.txt, sitemap.xml and schema.org share one property: platforms publicly committed to reading them first, then site owners followed. llms.txt inverted that — owners have published for eighteen months and no major AI platform has publicly committed to reading it. Google itself is split. The Search docs state in the Mythbusting section that you don't need new machine-readable files, AI text files, markup or Markdown. Meanwhile Chrome shipped the Agentic Browsing audits in Lighthouse's default config, one of which checks your root for exactly that file.

That collision lands on small businesses and independent studios. Next week a client will arrive with a PageSpeed Insights report asking why their Agentic Browsing score is 2/3. This piece answers three things: whether that point is worth fixing, who would read the file if you fixed it, and what the layer beyond llms.txt actually requires — with hours and costs.

Timeline and the full numbers

July 2025: Google's Gary Illyes says Google does not support it and has no plans to.
7 Nov 2025: SE Ranking publishes its 300,000-domain study — zero citation correlation.
7 May 2026: Lighthouse 13.3.0 ships Agentic Browsing in the default config, putting the llms.txt check into every PageSpeed Insights run.
Mid-May 2026: Google Search publishes its AI optimisation guide saying these files aren't needed. Less than a week later, Chrome begins auditing for one.
15 Jun 2026: Ahrefs publishes server-log analysis of 137,210 domains.
18 Jun 2026: Lighthouse issue #17082 — a file validated with zero errors across 882 links in 18 sections is reported as "fetch failed" while CDN logs show Google IPs fetching it successfully twice. Labelled P1.
18 Sep 2026: Mueller flags the directory-site chain effect; Barry Schwartz folds it into that day's Search Engine Roundtable recap.

The Ahrefs log numbers are the brutal part. Of 137,210 domains, 28% (about 38,360) publish a valid file, and 97% of those received zero requests across all of May 2026. The remaining 3% absorbed all ~22,000 measured requests, 96% of them from bots: SEO audit tools 21.7%, unidentified traffic 14.9%, general crawlers 13.1%, tech-profiling tools 11.6%. AI retrieval bots — OAI-SearchBot, PerplexityBot, the ones that decide whether you get cited — accounted for 1.1%, with Slackbot's link previews outnumbering PerplexityBot. The closing argument is the 404 data: across every request to a non-existent llms.txt, the AI-bot share was zero. No AI system knocks on a door that isn't there, so "skip it and you go invisible" was never true.

What each reader should do now

Small business owners

When an agency quotes off an Agentic Browsing score, ask first: "In the last 30 days, how many times did OAI-SearchBot or PerplexityBot fetch my /llms.txt?" No answer, no payment.
Move that budget to first-party substance: price ranges, lead times, service scope, real case numbers. That's what gets pulled when you're cited.

Studios and developers

Agentic Browsing checks llms.txt, WebMCP and accessibility-tree integrity, plus CLS. The last three have real technical depth and help human users too — that's the billable work.
Google's docs say agents treat the accessibility tree as their primary data model. Adding programmatic labels and un-hiding wrongly buried interactive content improves agent usability and genuine accessibility compliance at once.
If you do ship an llms.txt, manage it like code: version control, restricted edit rights, links only to resources you control. Ahrefs found a crawler calling itself prompt-injection-survey — someone is already studying the file as an injection surface.

Content marketers

Stop reporting "llms.txt is live" as an outcome. Report "pages cited by AI this month" and what those pages have in common.
The top AI consumer of the file is agents and agentic infrastructure (10.5%), within which Claude-Code out-fetched every AI retrieval bot. Developer tools, APIs and SaaS docs get value here. Restaurants, interiors, aesthetics and ecommerce do not.

Impact comparison

Who	Next 30 days	Do this	Not this
SMB	Vendor upsells off a 2/3 screenshot	Demand log evidence	Fund a project for one audit item
Freelance studio	Clients want "AI optimisation," undefined	Sell a11y-tree repair, schema, log monitoring	Sell llms.txt authoring as the headline
Content marketing	AI-visibility KPI loses its basis	Switch to cited-page counts	Count deployment as an outcome
Devtools / docs sites	Coding-agent fetches keep rising	Maintain Markdown under /docs	Market it as an SEO tactic
Ecommerce / local	Effectively no change	Add first-party detail and schema.org	Spend on machine-readable files

Who sells "AI visibility auditing"

Tool	Core function	Price	Best for
Chrome Lighthouse / PSI	Three Agentic Browsing audits plus CLS	Free	Everyone — watch the #17082 false negative
Ahrefs Bot Analytics	Real crawler requests split by user agent	Paid subscription	Teams needing evidence to rebut "AI reads it"
SE Ranking / SE Visible	AI citation tracking, cross-platform visibility	Paid subscription	Agencies that owe monthly reporting
Profound and similar AEO platforms	Brand mention monitoring across AI platforms	Paid (enterprise tier)	Clients with brand-tracking budget
Self-hosted log parsing	AI bot behaviour from Nginx / Cloudflare logs	~NT$0 in licences	Taiwanese SMBs and studios

What nobody will tell you

One: Google's "contradiction" isn't one — and it hurts anyway. The Search docs describe being found; Lighthouse describes being operated by an agent. Technically compatible. The problem is nobody explains the distinction to a small business owner. They see a Google tool showing red and pay to make it go away. Demand manufactured by interface design.

Two: the audit itself may be wrong. The reporter of issue #17082 did everything right and still failed. More absurdly, PSI ran HeadlessChromium 146 while Google's own scoring docs state the category requires Chrome 150 or later. You may be paying to satisfy an audit running unsupported.

Three: "no harm in having it" isn't true. Ahrefs calls it a security risk: agents are built to trust this file, and research crawlers are already probing it as a prompt-injection surface. An llms.txt nobody maintains, gone stale or silently rewritten by a plugin, misleads every agent that reads it.

Four: 12% of requests are the industry studying itself. GEO/AEO tools 5.8%, directories and validators 3.6%, research crawlers 2.7%. Add the directory-site chain effect and much of "someone is reading it" is an echo this ecosystem produced on its own.

The zero-subscription route for Taiwanese SMBs and studios

Instead of buying an AI-visibility subscription, read your own logs. This works on any Laravel site or Nginx / Cloudflare setup at NT$0 in licence fees.

Step 1 (0.5 h): grep -Ei "GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot|Claude-Code" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50 — shows which pages AI actually reads.
Step 2 (1 h): wrap it in a daily cron writing to a table; expose a 30-day trend via Laravel's scheduler and one Blade report page.
Step 3 (2 h): run Lighthouse and fix only the accessibility tree and CLS — both help humans and agents, and neither depends on a platform promise.
Step 4 (1 h): turn price ranges, scope, lead times and case numbers into independently quotable passages with Organization and FAQPage structured data.

Total: about 4.5 hours. At the NT$1,200–2,000 per hour typical of Taiwan's contract market, that's a one-off NT$5,400–9,000, with under half an hour of monthly upkeep. Against subscriptions running thousands to tens of thousands of NT dollars monthly, it pays for itself in month two — and the data is yours.

30 / 60 / 90-day checkpoints

☐ Day 30: log report live, listing fetch counts and top pages per AI bot
☐ Day 30: confirm actual request count on /llms.txt; if zero, drop the item from proposals and reports
☐ Day 60: Lighthouse accessibility-tree and CLS passing, with before/after evidence
☐ Day 60: first-party passages published; Rich Results Test clean
☐ Day 90: confirm the new passages were fetched by AI retrieval bots
☐ Day 90: compute overlap between pages AI bots fetched and pages enquiries came from

FAQ

I already have an llms.txt. Should I delete it?

No, but change how you manage it: version control, restricted edit rights, content limited to links and plain descriptions, links only to resources you control, and an alert on unauthorised changes. The real risk isn't that the file exists — it's that it sits unmaintained and stale while being read by agents designed to trust it.

Agentic Browsing shows 2/3. Does that hurt my Google rankings?

No. Google Search's documentation states plainly in its mythbusting section that you don't need machine-readable files of this kind to appear in generative AI search. Agentic Browsing is experimental and doesn't even produce a 0–100 score, only a pass ratio. It measures agent operability, not search ranking.

So should I build an llms.txt or not?

It depends who your customers are. Developer tooling, API services and technical documentation sites have support in the data — coding agents fetch these files more than every AI retrieval bot combined. Food service, interiors, aesthetics, local services and general ecommerce should expect the 97% zero-request base rate. If your CMS generates one automatically, leave it and spend nothing further.

What's more worth doing than llms.txt?

Plenty, in a clear order: write first-party information as independently quotable passages, add schema.org structured data, fix the accessibility tree and layout stability, and only then consider machine-readable files. The first three help humans and machines alike and depend on no future platform commitment.

My take

The narrative has split into two camps: llms.txt is dead, or llms.txt is early positioning for the agentic web. I'm in neither. My call: llms.txt survives, but as an agent interface for developer documentation — and that version has nothing to do with SEO. What actually gets cleared out within eighteen months is the entire "AI visibility auditing" tool category. Once Chrome puts Agentic Browsing into every free PageSpeed Insights run, tools charging monthly to scan your site and hand you an AI-readiness score have had their core feature absorbed by a browser built-in.

For a Laravel + Flutter contract studio like ScriptWalker, the implication is blunt: don't productise "llms.txt authoring" — productise the agent-readable interface layer. The former is a Markdown file a client generates with a plugin in five minutes, margin near zero. The latter covers accessibility-tree repair, semantic HTML refactoring, structured data, WebMCP endpoints and that self-hosted log dashboard — each requiring real understanding of front-end rendering and back-end routing, which SEO shops can't deliver and which produces value for human users too. When a client arrives holding a red report, what you sell them isn't a file. It's the ability to tell a real signal from tool noise.

Sources

First-party

Third-party and empirical studies