Short answer: for most SME websites, it takes days to weeks before AI re-reads changed content. Cloudflare's CEO revealed on the Q3 call that OpenAI now pays for "has this page changed" signals. SMEs should fix sitemap lastmod and turn on IndexNow; four free settings take about half a day.
On November 8, Search Engine Journal summarized Cloudflare's Q3 earnings call: CEO Matthew Prince said OpenAI pays Cloudflare for signals about which pages have changed, so its crawler stops re-fetching unchanged pages. He repeated that within five years AI agents could generate 1,000 times more traffic than humans, and called making a site easy for agents "the new version of SEO." Cloudflare's Q3 revenue was $782.5 million, up 39%. In one line: AI companies have started paying for freshness.
Why now? AI crawlers multiplied over the past year, but most fetches hit unchanged content; sites pay for bandwidth and AI firms pay for compute. Cloudflare and OpenAI began a research pilot in July, and in October Cloudflare said a cohort of about 1% of its network helped OpenAI discover and index hundreds of millions of new pages. Prince's example is close to home for SMEs: a user looks at one restaurant, but an agent may pull menus from 1,000 sites, leaving 999 to bear the cost. Cloudflare added more than 158,000 paying customers this quarter; pricing power over agent traffic is concentrating at the CDN layer.
Players are taking different routes. Cloudflare started with blocking and now offers per-request pricing through Monetization Gateway and Pay Per Use (in beta since late September). TollBit focuses on charging AI companies on behalf of publishers. Microsoft made Clarity's agent activity dashboard generally available in late October so sites can see which agents visited and what they did. The category is moving from "block or not" to "how to price and how to notify," and whoever controls change signals decides what AI reads first.
For a Taiwan SME, the risk is simple: you change a price, opening hours or a promotion, and ChatGPT or Copilot keeps quoting the old one. Below: the details, actions for three kinds of readers, a tool comparison, and settings that cost nothing.
The details: AI companies now pay for page freshness
The crawler cost structure has changed, and telling crawlers which pages changed now has commercial value. Key points from the call and official materials:
- OpenAI pays for Cloudflare's page-change signals; Prince expects other AI companies to follow.
- Cloudflare booked its first "Act 4" revenue, its new business model for the agent era, without disclosing the amount.
- Prince projects nonhuman traffic could reach 1,000 times human traffic within five years, sooner than he said in August.
- Agents that state their purpose may get lower access prices; a Hawaii trip-planning request that creates ad opportunities might even be free.
- AI Crawl Control is available on every plan and lets sites allow or deny individual AI crawlers.
Winners are sites that update often and are easy to parse; losers are sites with prices in images and sitemap dates that are wrong.
What three kinds of readers should do now
Bottom line: tell crawlers which pages changed before worrying about blocking or charging them.
- Brand owners / SME owners: list the 10 pages that change most (prices, hours, promotions, stock), make sure the text is real text, not images, and notify crawlers after each change.
- Marketing / SEO operators: turn on IndexNow in Bing Webmaster Tools, check that sitemap last-modified dates in Search Console are accurate, and review AI crawler logs monthly.
- Developers / agencies: update lastmod only when content really changes, send IndexNow automatically on save, and enable Crawler Hints for sites on Cloudflare.
Tool comparison: change notification and AI crawler management
For SMEs, free IndexNow and Crawler Hints are enough; paid options are for large publishers.
| Tool | What it does | Cost | Best for |
|---|---|---|---|
| IndexNow (supported by Bing and others) | Notifies search engines when content changes | Free | Every site |
| Cloudflare Crawler Hints | Sends change notifications for you | Free plan works | Sites already on Cloudflare |
| Cloudflare AI Crawl Control | Allow or block AI crawlers, view logs | All plans; per-request pricing in beta | Sites that want crawler control |
| Microsoft Clarity agent activity | Shows which AI agents visited and what they did | Free | E-commerce and booking sites |
| TollBit | Charges AI companies for content use | Per agreement | Media and large content sites |
What they won't tell you
Counterpoint one: Google does not support IndexNow, and Google only uses lastmod when it is consistently and verifiably accurate. Fake sitemap dates teach Google to ignore your signal.
Counterpoint two: 1,000x is a forecast, not a statistic, from the CEO of a company that sells bandwidth and protection. SMEs do not need to buy protection plans because of that number.
Third, OpenAI and Cloudflare are the ones paying for change signals, not you. The signal format sits with the CDN; sites not on Cloudflare rely on IndexNow and sitemaps for now.
SMB alternative without SaaS subscriptions
Four free settings, about half a day of work, help AI read your latest content sooner.
- Fix the sitemap: update lastmod only when content changes, not to today's date every day.
- Turn on IndexNow: WordPress has plugins; a Laravel site can send one API request in its save event.
- Show prices and business info as text and structured data, not images.
- On Cloudflare's free plan, enable Crawler Hints and check AI Crawl Control so you are not accidentally blocking ChatGPT or Bing crawlers.
FAQ
Here are the 4 questions readers ask most; the first sentence of each answer is the answer.
How long until AI re-reads my site's updates?
There is no fixed time; days to weeks is common. Sites that update often, keep accurate sitemap dates and send IndexNow are usually re-read sooner.
What is IndexNow, and does Google support it?
IndexNow is a free protocol that notifies search engines when content changes. Bing and others support it; Google does not, so sitemap lastmod must stay accurate.
Should SMEs block AI crawlers?
Usually not entirely. Being read and recommended by AI is typically worth more than the bandwidth saved; block only abnormal or unidentified crawlers.
How much do these settings cost?
Most are free and take half a day to three days. WordPress needs a plugin; a custom site needs a save-event hook, about NT$5,000–20,000 if outsourced.
My take: freshness will be worth more than ranking
My view differs from the mainstream: everyone is chasing "will AI cite me," but over the next 18 months SMEs will lose most from AI citing stale data. Once prices, hours or promotion deadlines expire, the more AI cites you, the more complaints you get. Change notification will become basic website equipment like SSL, not an SEO bonus.
Another contrarian point: most SMEs should not rush to charge AI crawlers. You are not a news publisher; being read is worth far more than the cents you could collect.
For ScriptWalker (a Taiwan custom development studio working in Laravel and Flutter), the lesson: we will make "accurate lastmod, IndexNow on save, AI crawler log report" standard on new sites, and offer a one-time AI readability check for existing sites, about 1–3 days of work.
Sources
- Cloudflare Q3 2026 results release (SEC) (primary)
- Cloudflare blog: discovery pilot results with OpenAI (Oct 15, 2026) (primary)
- Cloudflare AI Crawl Control documentation (primary)
- Google Search Central: build and submit a sitemap (primary)
- IndexNow official site (primary)
- Search Engine Journal: Cloudflare CEO Likens AI Agent Optimization To A New Version Of SEO (Nov 8, 2026) (third party)
- Cloudflare Q3 2026 earnings call transcript summary (third party)