Tomorrow, on 15 September 2026, Cloudflare flips the default settings for AI crawlers. The switch was announced on 1 July at its second Content Independence Day, and the first-party notice sits in Cloudflare's official blog post "Your site, your rules": crawlers are split into Search, Agent and Training, and on pages that display ads, Training and Agent are blocked by default while Search stays allowed. The line that actually bites is the inconspicuous second one in the same post — multi-purpose crawlers are judged by all of their behaviours, under the most restrictive rule. TechCrunch's same-day report confirms the new defaults cover new customers, new sites added by existing customers, and all existing free-plan customers.
The timing is not a coincidence. Cloudflare co-founder Matthew Prince flagged a milestone: non-human traffic on the internet has overtaken human traffic for the first time, a year earlier than expected. The heart of the dispute is the gap between crawling and referral — over a 28-day window ending in late July 2026, Anthropic's ClaudeBot crawled roughly 2,237 pages for every visitor it referred, OpenAI's GPTBot around 217, and Google only 4.6. Uglier still, more than half of AI crawl traffic is spent re-fetching pages that have not changed. Pay Per Crawl, launched in July 2025, was the first attempt at charging; this July it evolved into Pay Per Use, billing not by how often content is fetched but by when it creates value, with Ceramic.ai and You.com as the first partners.
No peer is taking this route. AWS WAF Bot Control only separates Common from Targeted; Akamai Bot Manager and DataDome still organise around fraud and card abuse; Fastly keeps bot management inside enterprise contracts. Nobody else has turned crawler purpose into a network-wide default policy. Cloudflare can do it because it sits in front of roughly a fifth of the world's websites. The Content Signals Policy launched in September 2025 gained a fourth field this year, use, with values of immediate, reference and full — extending "may you crawl" into "how much may you keep".
A large share of Taiwanese SMB websites sit on Cloudflare's free plan, and most of those settings were ticked casually years ago. If you once enabled that reassuring-looking Block AI bots toggle, its effect after tomorrow is not protection — it is removing yourself from the AI index. Below is what actually changed, who gets hit, and what to do in the next 24 hours.
What Happened: Three Categories, One Strictest-Rule Clause, One Overlooked Side Effect
Cloudflare's category definitions are worth reading line by line, because each one carries a completely different commercial consequence.
- Search: collecting and indexing your content so it can answer questions about it later. Site owners can expect referral traffic or equitable compensation in return, so it stays allowed by default.
- Agent: acting in real time on behalf of a person, including chat fetch bots such as ChatGPT-User and browser-driving Gemini or Claude agents. The key point is that a human is waiting on the other end.
- Training: absorbing your content permanently into model weights.
Enterprise customers also get BotBase, which catalogues known bots across 11 behaviour classifications. But two sentences decide who lives and who dies. First, on ad-bearing pages the default becomes blocking Training and Agent, on the reasoning that an ad signals the page was meant for a human. Second, multi-purpose crawlers are enforced under the most restrictive rule, and the announcement names Googlebot, Applebot and BingBot directly: if you have ever selected to block Training, through the new menu or the legacy Block AI bots service, all three get blocked from tomorrow — and the legacy toggle itself is deprecated the same day. Each category offers three treatments: block site-wide, block only on pages with ads, or allow.
The meaning of Verified changes too: previously any Verified bot was allowed by default, now it only means the bot can be classified, and entry depends on which categories you allow. This is especially sensitive for Google — it offers Google-Extended so you can opt out of Gemini and Vertex training, but flagship Googlebot simultaneously serves Search, AI Overviews and AI Mode and cannot be split.
Immediate Actions for Three Types of Reader
Brand owners and SMB founders
- ☐ Log into Cloudflare today, open Configure AI bot policies under Security Settings, check the state of all three categories and screenshot it for the record.
- ☐ If your goal is to be recommended by ChatGPT, Gemini or Perplexity, allow both Search and Agent, and only consider blocking Training.
- ☐ Check content pages carrying AdSense especially closely — that is where the new defaults act first.
Marketers and SEO practitioners
- ☐ Mark 15 September as the watershed, and cross-check GA4's AI Assistant channel and Search Console's generative AI reports for a break in the curve over the following two weeks.
- ☐ State your position in robots.txt using all four Content Signals fields, especially use=reference, meaning indexing and excerpting are fine but link back.
- ☐ Maintain a written allow-list of which bots your site permits, and re-verify it after every rebuild.
Developers and agencies
- ☐ Pull bot User-Agents from nginx access logs or Cloudflare Logpush and compute a crawl-to-refer ratio per bot.
- ☐ Check whether the client's CDN configuration also covers staging and subdomains; disasters usually happen on an unattended blog subdomain.
- ☐ Use the Laravel scheduler to diff robots.txt and bot settings daily, and email an alert on any change.
AI Crawler Control Tools Compared
Start tool selection with one question: can it configure Search, Agent and Training separately? If it cannot, you are left with two bad options — block everything or allow everything, and both are wrong for a company that wants AI to recommend it.
| Option | Separates Search / Agent / Training? | Public pricing (official pricing pages govern) | Best for |
|---|---|---|---|
| Cloudflare Free | Yes, all three configurable independently | US$0 | Single-site SMBs and blogs |
| Cloudflare Pro / Business | Yes, plus Super Bot Fight Mode and Bot Analytics | About US$25 / US$250 per domain per month | Mid-size sites with commerce or forms |
| AWS WAF Bot Control | No, only Common and Targeted | US$10 per web ACL per month plus US$1 per million requests | Stacks already on AWS |
| DataDome | No, focused on fraud and card abuse | From roughly US$1,590 per month | High-value transactional sites |
| Self-hosted nginx / Laravel middleware | Yes, but you maintain the rules | Server cost only | Agencies and internal platforms |
What They Will Not Tell You
One, this is not a win for small businesses; it is a distribution tax. Only publishers with volume get a seat at the Pay Per Use negotiating table. A Taiwanese service company's site with three thousand monthly visits will receive exactly zero in licensing revenue while losing very real AI exposure.
Two, blocking Agent is the easiest way to wound yourself. Cloudflare's own definition says it: on the other end of an Agent request, a real person is waiting for the result — the highest-intent traffic you will ever get. The logic of blocking Agent on ad pages was designed for publishers' display-ad model, and simply does not hold for a company that lives on inbound enquiry forms.
Three, Cloudflare is not a neutral third party. It is simultaneously the traffic gatekeeper, the operator of the paid marketplace, and a vendor of its own AI products. Turning purpose classification into a network-wide default makes it the unavoidable toll booth.
No SaaS Subscription: A Three-Stage SMB Rescue Plan
- Stage one (today and tomorrow), stop the bleeding: log into Cloudflare and confirm all three category settings. If AI exposure is the goal, allow Search and Agent and decide Training on your own terms. Explicitly mark in Security settings whether the new defaults apply — do not let the system decide for you.
- Stage two (week one), state your position: add the four Content Signals fields to robots.txt. The common combination allows search, disallows ai-train and marks use=reference, and the same declaration also covers subdomains that do not sit behind Cloudflare. Zero cost, cross-platform, vendor-neutral.
- Stage three (weeks two to four), measure: write a script that parses access logs daily, counts requests from GPTBot, ClaudeBot, PerplexityBot and Googlebot, and compares them against sessions in GA4's AI Assistant channel to derive your own crawl-to-refer ratio. Only with that curve can you tell whether blocking or allowing made you money.
Frequently Asked Questions
I do not run ads. Am I unaffected?
Ad pages are only where the new defaults act first. The clause that applies site-wide is that multi-purpose crawlers are enforced under the most restrictive rule. If you have ever ticked to block Training, Googlebot, BingBot and Applebot get blocked along with it, ads or no ads.
Will blocking Training stop ChatGPT from citing me?
Not necessarily, but the risk is high. Training and live retrieval are theoretically separate paths; the problem is that many model providers still run one crawler for several purposes, and the strictest-rule verdict means you lose retrieval as well.
How do I check whether I am affected?
Open Configure AI bot policies under Security Settings and read the state of the three categories, then check whether the legacy Block AI bots toggle is on. Check both, because the legacy switch is being reinterpreted after tomorrow.
Can an SMB realistically earn money from Pay Per Use?
Almost certainly not in the near term. The first partners are only Ceramic.ai and You.com, and both the settlement thresholds and the content volumes point at publishers. Watch it as a long-term option, but do not sacrifice today's exposure for it.
My Take
My judgement runs against the current consensus: for 99% of Taiwanese SMB websites, blocking AI crawlers is a negative-ROI decision, and the real risk over the next 12 months is not that your content gets taken, but that a default quietly deletes you from the AI index. Publishers sell the content itself, so an excerpt is a loss. You sell services and delivery capability; content is customer-acquisition material, and one citation is worth far more than ten fewer pages crawled. My other unpopular call: Pay Per Use will not put a meaningful payment into any SMB's bank account within 18 months. What Cloudflare is really selling is a visibility control panel.
For a Laravel plus Flutter team like ScriptWalker, the opportunity sits between panic and ignorance, and packages into two deliverables. First, an AI crawler audit and governance project — a one-off engagement inventorying CDN settings, robots.txt, Content Signals and subdomains, producing an allow-list document, well suited to group clients running 3 to 20 properties. Second, a crawler traffic dashboard — parse logs and Logpush in Laravel, cross-reference each bot against GA4's AI Assistant channel, and turn the crawl-to-refer ratio into a monthly report. Tool vendors sell the switch; agencies sell knowing which way to flip it.
Sources
- First-party: Cloudflare — Your site, your rules: new AI traffic options for all customers (2026-07-01)
- First-party: Cloudflare — Content Independence Day, one year on: building the business model for the agentic Internet
- First-party: Cloudflare — Content Signals Policy, contentsignals.org
- First-party: Cloudflare Docs — Block AI bots, BotBase, Verified bots
- First-party: Google Search Central — Google common crawlers (including Google-Extended)
- First-party: Cloudflare — A deeper look at AI crawlers: traffic by purpose and industry
- Third-party: TechCrunch — Cloudflare's new policy pushes AI companies to pay for publishers' content (2026-07-01)
- Third-party: The Next Web — Cloudflare gives AI crawlers a September deadline
- Third-party: AWS WAF Pricing, Cloudflare Plans