AI crawls your site ten thousand times to send back one visitor
In June 2026, Cloudflare's latest crawl-to-refer ratio on Radar put a bill most SMBs never tallied on the table: in the week of May 25 to June 1, Anthropic's ClaudeBot fetched on average 11,122 pages for every 1 visitor it sent back; OpenAI was 857:1, Perplexity 190:1, and traditional Googlebot just 5:1. In one sentence — AI engines take your content far faster than they send people back to it.
To weigh that number, set it in the last 18 months of market structure. Since the "answer layer" emerged in late 2024, AI engines crawl the web with two kinds of bots: training crawlers that ingest your content into a model, and search crawlers that fetch in real time when a user asks a question. Cloudflare's data shows training crawls still outnumber search crawls by more than 5 to 1, with search just 9.3% of AI crawler traffic (up from 7.5% in April). In other words, AI's relationship to web content is still mostly extractive, not referral-driving.
The peer comparison stings more. Googlebot fetches 5 pages per visitor returned because traditional search is built on sending traffic back. AI engines have no such constraint — they write the answer straight into the chat box, so the user never clicks through. Anthropic's ratio improved from 13,528:1 in April to 11,122:1 but is still the worst of any major operator; OpenAI went from 1,252:1 to 857:1; Perplexity moved the wrong way, from 95:1 to 190:1. The category's direction: crawl harder, cite pickier.
What does this mean for SMBs? Every AI chatbot combined contributes just 0.29% of all search referral traffic. This article covers how the ratio hits your bandwidth bill and your GEO strategy, which tools measure your own number, and why "just block the AI crawlers" is wrong for most small businesses.
Event detail and the full numbers
Three numbers matter most. First, the spread is enormous: Googlebot 5:1, Perplexity 190:1, OpenAI 857:1, ClaudeBot 11,122:1 — the same single crawl can differ two-thousand-fold in whether it returns a visitor. Second, the trend is worsening: Perplexity went 95:1 to 190:1 in a month, crawling more and returning less. Third, AI referrals are only 0.29% of all search referral traffic, so against the "AI will replace Google traffic" narrative, the actual referral contribution to SMBs is still small. Who benefits? Sites cited as authoritative sources. Who loses? Sites whose content is ingested for training but get no return — the situation for most SMBs.
Immediate actions for three readers
- Brand owners / SMB founders: before asking "should we do GEO," measure whether AI returns anyone. Have someone pull 90 days of sessions from chatgpt.com / perplexity.ai / gemini.google.com referrals in GA4 — that number decides how much to invest.
- Marketers / SEO operators: structure content (clear question-style headings, extractable paragraphs, FAQ schema) to be cited, not clicked; in Cloudflare, allow search crawlers and rate-limit pure training crawlers.
- Developers / agencies: export the client's server logs and compute "fetches divided by visitors returned" per AI bot. That report is itself a billable health check.
SaaS tool comparison
| Tool | Function | Price tier | Best for |
|---|---|---|---|
| Cloudflare (Radar / Bot mgmt) | See bot share, allow/limit crawlers | Free, up with plan | Control crawl cost while keeping visibility |
| Profound | Track brand citation across AI engines | Enterprise (high) | Budget for cross-engine monitoring at scale |
| Peec AI | GEO/AEO visibility tracking, competitor view | Mid | Marketing teams watching daily rank shifts |
| Own logs + GA4 | DIY crawl-to-refer and AI referral | Near zero | SMBs validating whether to pay at all |
What they won't tell you
- Counterpoint one: blocking crawlers = self-deletion. Plenty of consultants say "block AI crawlers to stop the bleed," but being fetched by search crawlers is your only entry into AI answers. A full block removes you from ChatGPT and Perplexity results — worse than being crawled.
- Counterpoint two: 0.29% is not "ignore it." The small overall share is an average dragged down by big sites. In some niches, AI citations bring small but high-intent, high-conversion traffic — low volume, high quality. Do not quit on the aggregate alone.
- Edge case: one user-agent may do both training and search fetches; a blunt block hurts the half that would have cited you.
The no-SaaS SMB alternative
- Use Cloudflare's free tier to see bot share — who crawls you and how hard.
- Parse your own server logs (a grep plus a spreadsheet): divide each AI bot's fetch count by that referral's GA4 sessions to get your own crawl-to-refer ratio.
- Tier with robots.txt + Cloudflare rules: rate-limit or block declared training crawlers, allow search crawlers, review monthly.
- On content: add FAQ schema and put the answer in the first two sentences of a paragraph to raise your odds of being extracted as the answer — more direct than buying a tool.
FAQ
Should I just block AI crawlers?
For most SMBs, no. Being crawled is your only path into ChatGPT and Perplexity answers; a full block deletes you from AI search. Tier it — block pure training crawlers, keep the real-time search crawlers that may cite you.
Will all that crawling crush my server?
It depends. A small host hammered by a high-frequency crawler (ClaudeBot's 11,122:1) on static pages will feel it. Check bot share on Cloudflare's free tier, then rate-limit training crawlers if needed.
How do I know if AI sends traffic back?
Check GA4 referrals for chatgpt.com, perplexity.ai, gemini.google.com and compare with bot fetches in your logs. Fetches divided by visitors is your own crawl-to-refer ratio.
Do SMBs need a paid GEO tool?
Not at first. Cloudflare Radar data, your logs, and GA4 referrals cover ~80%. Pay only when tracking dozens of keywords across multiple engines at scale.
My take
The mainstream splits two ways: "AI is stealing your content, block it," and "buy a GEO tool to win citations." My read differs from both: for SMBs, the crawl-to-refer ratio really tells you to spend on being cited, not clicked. A figure like 11,122:1 will become a content-licensing bargaining chip over the next 12 months — large publishers will use it to charge AI companies, but SMBs have no negotiating table, and a hard block only marginalizes them. The ScriptWalker takeaway is concrete: package an "AI crawl cost + visibility health check" as a standardized small service — pull a client's server logs and Cloudflare data, compute their own crawl-to-refer ratio, flag bots that only extract, and hand over a tiered allow/limit rule set. It ships from public data, the value is legible to the client, and it makes a clean entry engagement that can lead to a long-term retainer.
Sources
- Cloudflare Radar blog — AI search crawl-to-refer ratio: https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/
- Cloudflare blog — The crawl-to-click gap: https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Cloudflare blog — AI crawler traffic by purpose and industry: https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/
- Cloudflare Radar (public data platform): https://radar.cloudflare.com/
- Google Search Central — structured data and AI features: https://developers.google.com/search/docs