Opening narrative + deep analysis
In early July, Cloudflare dropped a policy that directly changes whether "your content can be seen by AI": starting September 15, 2026, it will by default block "mixed-use" AI crawlers from pages that carry ads. The first-hand report is from TechCrunch (July 1). "Mixed-use" means AI companies that blend search, agent, and training in one crawler — after 9/15, such crawlers get no access to ad-carrying pages unless a site explicitly opts them in.
To see why now, look at the last 12 months of market structure. The AI answer layer is shifting traffic from "click into the site" to "read it in the answer and leave" — Gartner expects traditional search volume to fall roughly 25% by 2026, and ChatGPT alone serves over 900 million people a week. Content scraped to feed models while sites get zero clicks — that "free lunch" tension built for a year, and Cloudflare, one of the world''s largest CDNs, holds a gate it can shut for content owners. The change isn''t "one more toggle"; it flips the default from allow to block.
Alongside it, Cloudflare upgraded its Pay Per Crawl marketplace to Pay Per Use — letting content owners charge when their content generates value for AI, not merely when it is fetched. Initial partners include Ceramic.ai and You.com. It even named the "world''s largest search engine" (Google) as holding about "2x more information" than other AI companies. The whole category is moving from "free scraping" to "tiered billing."
Will SMBs be affected? Yes — possibly in the opposite direction from what you''d expect. Below I unpack what the policy actually changes, what three roles should do, how monitoring tools compare, and how to protect yourself without a SaaS subscription.
Event details + full numbers
The scope of the default change is key: new Cloudflare customers, new sites created by existing customers, and all existing free-tier customers auto-apply "block mixed-use crawlers on ad pages" from 9/15. Existing paying customers keep override rights and can re-admit specific crawlers from the dashboard. In other words: if you''re an SMB running your site on Cloudflare''s free tier, this default lands on you directly; if you want certain AI engines to read you, you may need to check the setting yourself.
Cloudflare''s reasoning: most owners want content discoverable via search, often via AI too, but don''t want to hand over IP for free. The policy aims to force AI companies to separate search crawlers from training crawlers — you can be indexed, just not have the same crawler quietly haul content off to training. For content owners, it''s the first infrastructure-level tool to draw a line between "seen by AI" and "taken by AI for free."
Immediate actions for three readers
- Brand owners / SMB bosses: confirm whether your site runs on Cloudflare and on the free tier. If so, before 9/15 check Bot / AI-crawler settings and decide whether you want "cited by AI" or "block AI freeloading" — a business choice, not a technical detail.
- Marketing / SEO operators: add "AI-crawler visibility" to your monthly report. Use server logs or Cloudflare analytics to see which AI bots (GPTBot, PerplexityBot, Google-Extended) crawl you; don''t let a silent default block delist you from AI answers.
- Developers / agencies: audit clients''
robots.txt,llms.txt, and CDN bot rules for consistency. This becomes a billable "AI-visibility check-up" mini-service.
SaaS tool comparison
| Tool | What it does | Price band | Best for |
|---|---|---|---|
| Cloudflare (Bot mgmt / Pay Per Crawl) | Control which AI crawlers get access at the CDN layer; can set pricing | Free tier up; advanced tied to paid plans | Sites already on Cloudflare wanting to gate at infra level |
| Profound | Monitors brand appearance rate and citation sources in AI answers | Enterprise monthly (from hundreds USD) | Funded brands quantifying AI visibility |
| Scrunch AI | Tracks brand citations and sentiment across AI engines | Mid–high monthly | Teams comparing ChatGPT/Perplexity |
| Evertune | Analyzes which brands models recommend in your category | Enterprise | Category share-of-voice analysis |
What they won''t tell you
- "Block AI freeloading" and "be cited by AI" are often two sides of one switch: many engines'' real-time retrieval crawlers are the same as, or the same class as, training ones. A blanket block can also cut off Perplexity and ChatGPT search''s live citations. Cloudflare''s split logic helps, but in practice the line isn''t clean.
- The default hurts the uninformed: the ones to worry about aren''t big publishers (they''ll tune settings) but SMBs on the free tier who never look at bot reports — a switch they never touched could make them vanish from some AI answers after 9/15 without them noticing.
SMB alternatives without a SaaS subscription
You don''t need an enterprise subscription to protect yourself. A three-step DIY:
- Control crawlers yourself: in
robots.txt, explicitly allow/deny AI bots (GPTBot, PerplexityBot, Google-Extended, CCBot), and add anllms.txtstating your key content and licensing stance. - Read your own logs: use Nginx access logs plus a
grepto tally each AI user-agent''s crawl frequency — know who''s scraping you without buying monitoring SaaS. - Verify exposure yourself: each week, manually ask ChatGPT/Perplexity a few category questions and record whether you''re cited — the crudest but most honest AI-visibility tracking.
FAQ
I don''t run ads — does this affect me?
This default block targets ad-carrying pages; without ads the impact is smaller. But the overall tightening of bot policy is worth watching — still confirm your robots/bot settings once.
If I block AI crawlers, will my Google SEO suffer?
Depends who you block. Googlebot (search) and Google-Extended (AI training) are separate; blocking the latter doesn''t affect indexing. The key is not to conflate them and block everything at once.
Should an SMB block or allow?
No universal answer — it depends on your model: if content drives traffic and you want AI citations to convert, lean toward allowing real-time retrieval crawlers; if content itself is the product (paid reports), lean toward blocking. Decide whether being read by AI is an asset or a loss.
Can I change it after 9/15?
Yes. Paying customers can override anytime in the dashboard; free customers can adjust their settings too. It isn''t irreversible, but the default already made a choice for you — go confirm it''s the one you want.
My take
The mainstream reads this as "content owners finally won, AI has to pay." My call: it''s good news for big publishers and a new invisible risk for SMBs with no bargaining power. When "block" becomes the default, the most exposed are small sites that don''t even know who''s scraping them — they won''t collect Pay Per Crawl money, they''ll just find inquiries mysteriously dropping one day. The real structural shift isn''t "AI pays," it''s "AI visibility now requires active management," just as SEO went from "happens naturally" to "someone must own it" a decade ago. The lesson for ScriptWalker is direct: package an "AI-crawler visibility check-up" as a deliverable, billable service — audit robots.txt/llms.txt/CDN bot rules and produce a report on "which AI reads you today, which is blocked." One of the easiest-to-sell, easiest-to-explain new services of late 2026.
Sources
- TechCrunch: Cloudflare''s policy pushing AI to pay (first-hand, 2026-07-01) — https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/
- The Register: Cloudflare to block search-and-scrape bots — https://www.theregister.com/ai-and-ml/2026/07/01/cloudflare-to-block-cynical-search-and-scrape-bots-from-ad-supported-web-pages/5264727
- Cloudflare official blog — https://blog.cloudflare.com/
- The AI Insider: Cloudflare September deadline — https://theaiinsider.tech/2026/07/02/cloudflare-sets-september-deadline-to-force-ai-crawlers-apart-from-search-bots/
- Dataconomy: Cloudflare will block AI crawlers unless opt-in — https://dataconomy.com/2026/07/03/cloudflare-will-block-ai-crawlers-unless-sites-opt-in/