Cloudflare's Sept 15 AI Crawler Flip: Block Training, and You Block Googlebot Too

In 21 days your site gets a new crawler policy without you touching a single button

Twenty-one days to go. On September 15, 2026, Cloudflare flips the default settings for AI crawlers. Per its official blog announcement, bots are no longer sorted into "AI" and "non-AI" but by purpose: Search, Agent, and Training. From that date, on any page that displays ads, Training and Agent are blocked by default while Search stays allowed. The scope is the part that matters: the new defaults apply automatically to new customers, to new sites created by existing customers, and to every existing free-tier user. Paid customers with configured sites keep what they have.

This did not come out of nowhere. Over 18 months the terms of trade between content supply and AI demand were rewritten twice: first at Cloudflare's Content Independence Day in 2025, when new sites started defaulting to blocked alongside a pay-per-crawl marketplace, then in July 2026 when classification moved from identity to purpose (TechCrunch coverage). Why now? The traffic math stopped working. Research puts Google zero-click searches at 68% in early 2026, and on queries where an AI Overview appears, organic CTR falls 61% and paid CTR falls 68% — yet brands cited inside the Overview take 35% more organic clicks. Consumption is exploding, return traffic is collapsing, so the infrastructure layer took a seat at the negotiating table on content's behalf.

Follow the money and the picture sharpens: the measurement side is consolidating. Profound closed a $96M Series C at a $1B valuation; Scrunch AI was acquired by Sitecore on June 3, 2026 for a reported $225M; Evertune was acquired in December 2025. The direction is consistent. Measuring your visibility inside AI is being rolled up and sold as a subscription, while the switch that decides whether AI can read you at all sits with your CDN — and costs nothing.

Will SMBs feel this? Yes, in the easiest way to misread. An owner sees a harmless-looking "block AI training crawlers" toggle in a dashboard, flips it, and three months later watches traffic sag with no diagnosable cause. Below: what actually changes on 9/15, which sites are affected, and how to verify it without buying anything.

The event in full, with the numbers

• Effective date: September 15, 2026.
• New taxonomy: Search, Agent, and Training get separate policies instead of one blanket "AI bot" rule.
• New defaults: on pages that display ads, Training and Agent are blocked, Search is allowed.
• Who is affected: new customers, new sites created by existing customers, and all existing free-tier customers; paid customers are not switched automatically.

The most overlooked and most damaging detail is the collateral effect: if you block Training, multi-purpose crawlers such as Googlebot, Applebot, and BingBot get blocked too — even when Search is allowed. This is not a precision dial for site owners. It is leverage designed to force AI companies to split training and search crawlers into distinct user agents.

The heaviest losers are small businesses using Cloudflare as a free CDN with nobody reading crawl logs. Worse, the diagnostic tooling is shaky too: Google Search Console lost two full days of crawl stats on August 15 and 16.

What each reader should do this week

Brand owners and SMB operators

• Confirm two things: is your site on Cloudflare, and is it free tier? Those answers decide whether 9/15 touches you.
• Check whether your pages carry ad units. A pure marketing site with no ads is outside the default block.
• Screenshot your current bot settings before September 1. That is your only control group.

Marketing and SEO practitioners

• Build a GA4 "AI sources" segment now (chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com). You need two weeks of baseline before 9/15.
• Treat AI citations as their own KPI. Ahrefs' study of 3,000 websites found AI-sourced visitors were 0.5% of traffic but drove 12.1% of signups — a 23x higher conversion rate than organic search. Judge it by traffic share and you will undervalue it.
• Links and content that only exist after JavaScript renders are effectively invisible to most AI crawlers.

Developers and agencies

• Run a bot policy audit on every client site: Cloudflare settings, robots.txt, server UA rules, and WAF rules contradict each other more often than not.
• Pull 30 days of access logs, group by user agent, and produce a baseline of who crawled and which pages.
• Add a CI check that alerts on robots.txt or Cloudflare config drift — a common invisible incident in client work.

Crawler control vs. AI visibility tooling

The market splits into two layers: things that control access, and things that measure outcomes. Most buyers pay for the wrong layer.

What nobody will tell you

• "Charging AI companies" is a fiction for small businesses. Pay-per-crawl leverage depends on scale and scarcity. Your product pages, service pages, and blog posts are highly substitutable to a model, so their marginal price approaches zero. Blocking does not earn a licensing fee; it earns absence.
• The collateral blocking is a feature, not a bug. Blocking Training also blocks Googlebot, Applebot, and BingBot by design. The pressure is aimed at AI companies, but the cost lands on whoever flips the switch, and no UI will tell you that you are about to exit the index.
• The panic is also overblown in the other direction. The default block only applies to pages that display ads. Most SMB marketing sites carry no ad units and will see nothing happen on 9/15. The sites to actually check are content sites, media sites, and company blogs that bolted on AdSense.

The no-subscription DIY route for SMBs

• ☐ Pull the last 30 days of access logs and group by user agent with one awk/grep line: count pages fetched by GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended.
• ☐ Save it as baseline-2026-09-01.csv. That is your control group.
• ☐ Log into Cloudflare and screenshot the current bot/AI settings page, filed with the date.
• ☐ Check whether robots.txt conflicts with Cloudflare or WAF rules (classic failure: robots.txt allows, WAF blocks).
• ☐ Create a GA4 custom segment for referrals from the four major AI domains.
• ☐ Re-run the analysis on the 1st of each month and compare rates of change, not absolute numbers.
• ☐ Use an AI coding tool to turn it into a 30-line Python script on a cron job.

This will not tell you your share of voice inside ChatGPT answers. It answers the prerequisite question that matters more: is AI reading you at all?

FAQ

I'm on Cloudflare's free plan. Will my settings change on 9/15?

Yes. The new defaults apply to new customers, new sites created by existing customers, and all existing free-tier users; paid customers keep their configuration. If you do not want them, adjust the security settings before September 15.

If I block Training crawlers, will I disappear from Google Search?

Possibly. Googlebot, Applebot, and BingBot serve both search and training, so blocking Training blocks them too, even with Search allowed. Until AI companies split their crawlers, blocking carries far more risk than reward.

Should I block or allow AI crawlers?

One test: does your content have bargaining power? Content sites with proprietary databases, paywalls, and legal resources can reasonably block and negotiate; acquisition-driven brand sites and ecommerce stores should almost always allow.

How do I know whether AI is actually citing me?

Three layers: check server logs for AI crawler fetches, check GA4 for referrals from AI domains, and only then pay to query the models. Reversing that order wastes budget.

My take

The consensus reads 9/15 as a turning point for content monetization. I read it the opposite way: over the next 12 months, pay-per-crawl revenue to small businesses will round to zero, while the visibility they lose by blocking will be largely unmeasurable because it lands in direct traffic. You pay the cost and never see the invoice. The structural winner is the CDN, which aggregated millions of sites with no individual bargaining power into leverage of its own. Site owners got a toggle; Cloudflare got a seat at the table.

Second unpopular call: the AI visibility monitoring subscription model peaks before this crawler war resolves. When control is free and measurement costs thousands a year, the market eventually notices the ordering is wrong.

For an agency like ScriptWalker the implication is concrete: productize an "AI crawler access audit" as a fixed-scope, one-time deliverable. A forty-minute audit — Cloudflare config, robots.txt, WAF rules, JS rendering dependencies, 30-day log baseline — plus one comparison report. Clients want an answer, not another monthly dashboard.

Need help?

• Email: [email protected]
• Phone: 0916-224-047
• LINE: @ufv9089p

Sources

Primary

• Cloudflare blog: Your site, your rules: new AI traffic options for all customers
• Ahrefs research: AI traffic study (3,000 websites)
• Semrush newsroom: Expanded 2026 AI Visibility Index, analyzing 126 million AI search prompts

Third-party

• TechCrunch: Cloudflare's new policy pushes AI companies to pay for publishers' content
• Help Net Security: Cloudflare changes AI crawler access rules
• Search Engine Land: Google zero-click searches reach 68% in early 2026
• Search Engine Land: Google AI Overviews CTR shows early signs of recovery
• Search Engine Roundtable: Google Fixed Favicon Bug & Crawl Stats Missing Data