Short answer: Most SMBs shouldn't block them. On Oct 30, Google and Bing split "AI training" and "AI answer grounding" into separate switches, with Bing's taking effect on Feb 1, 2027. Keep both open for your marketing site and service pages, and block training only for paid courses or member content.
On October 30, Google announced on its Search Central blog that it will support a shared AI training preference being drafted at the IETF. With one shared token, __usage_ai-training, in robots.txt, a site can tell multiple AI companies not to train on its content; Google expects to support it in early 2027. The same day, Microsoft went further on the Bing blog, splitting training and grounding into two switches and honoring the new preferences from February 1, 2027. In short: whether AI learns from your content and whether it cites it in answers can now be managed separately.
Why now? Over the past 18 months, AI answers have absorbed a large share of clicks. The same week, Reddit's Q3 shareholder letter said search referrals fell by more than 1 million a week on average in October, and the CEO pointed to Google's AI Overviews and AI Mode. US logged-out daily users fell from 30.1 million to 27.3 million, and the stock dropped about a third. That pressure from large sites and publishers, including an EU antitrust complaint from French publishers, pushes search engines to offer finer controls before more sites block crawlers outright.
Approaches differ. Google previously had only Google-Extended, which covers Gemini training and grounding; OpenAI and Anthropic use their own crawler names. The shared token aims to unify training preferences, and Bing also makes grounding configurable. The category is moving from one rule per company toward a shared standard, but the IETF draft isn't final and participants still disagree.
For Taiwanese SMBs, the point is simple: your homepage, service pages and FAQs are content you want AI to cite. One wrong line could make ChatGPT, Copilot or Gemini miss you when someone asks where to get a service in Taipei. Below: how to write the settings, where the traps are, and how to manage it for free.
The details: training and grounding split, timeline in 2027
The core change is that training and grounding can be set separately, and Bing has published a clear timeline.
| Item | Microsoft Bing | |
|---|---|---|
| AI training | Shared token __usage_ai-training, expected early 2027 | Shared robots.txt token or page meta ai-training:disallow |
| AI grounding | Google-Extended governs grounding in Gemini apps | Page meta ai-grounding:disallow, or disallowextended for Microsoft-only use |
| Classic search ranking | No stated impact | Officially no change to Bing search eligibility |
| Timeline | Support expected early 2027 | Transition settings 2026/12/15, effective 2027/2/1, old directives deadline 2027/3/1 |
Bing warns that sites using old noarchive or nocache directives to limit AI must switch to the new preferences by March 1, 2027, or the restrictions lapse. Conversely, disallowing grounding will reduce brand visibility in AI answers.
What three kinds of readers should do now
Bottom line: owners should confirm nothing is blocked by accident, marketers should manage training and grounding separately, and developers should put robots.txt under version control.
- Brand owners and SMBs: Ask whoever runs your site one question: "Are we blocking any AI crawlers right now?" Plugins or hosting defaults often do it without the owner knowing.
- Marketers and SEOs: Split content in two. Content you want cited (services, price ranges, FAQs, case studies) stays open; only truly paid content (courses, reports, member areas) blocks training.
- Developers and agencies: Version-control robots.txt and log every change; record AI crawler visits in the back office so clients can see which pages AI reads.
A conservative example (block training only on a paid course path, keep everything else open):
User-Agent: __usage_ai-training
Disallow: /courses/paid/
User-Agent: Google-Extended
Allow: /Comparing the controls
Free robots.txt and meta tags are enough for SMBs; check CDN defaults carefully.
| Control | What it governs | Cost | When to use |
|---|---|---|---|
| Shared training token in robots.txt | Training preferences across AI providers | Free | Keep whole paths out of training |
| Page meta / X-Robots-Tag | Bing training and grounding | Free | Fine-grained page control |
| Google-Extended | Gemini training and grounding | Free | Signals to Google specifically |
| CDN or host AI bot switches | Blocks traffic outright | Usually included | Heavy scraping; check defaults first |
What the announcements won't tell you
The new standard looks tidy, but there are at least three traps.
- The Google-Extended trap: per Google, if you block training with the shared token and don't explicitly specify Google-Extended, Google treats it as if Google-Extended were disallowed, which can also cost you grounding in Gemini apps.
- The standard isn't final: the IETF draft is still under discussion, and participants dispute whether a user asking AI to read a page counts. Settings you write today may need changes.
- Blocking isn't retroactive: preferences only bind compliant crawlers and only apply going forward; content already used for training doesn't disappear.
A no-subscription alternative for SMBs
You don't need an AI visibility tool; four free actions cover it.
- Open your site's /robots.txt and confirm you aren't blocking Googlebot, Bingbot, GPTBot or other common crawlers.
- Turn on the AI Performance report in Bing Webmaster Tools to see which pages AI answers cite.
- Filter AI crawler user agents from your server logs and review pages and visit counts monthly.
- Rewrite service pages as "question headline + direct answer + price range" so AI is willing to cite them.
FAQ
Will blocking AI training hurt my Google rankings?
Officially, no impact on classic search rankings. Blocking grounding, however, reduces your chances of appearing in AI answers.
Should an SMB website block AI crawlers?
Mostly no. Your site exists to be found, and AI citations are extra exposure; block training only for paid content.
When do the new AI preferences take effect?
Bing honors them from February 1, 2027, and Google expects support in early 2027, so plan now.
How do I know whether AI cites my website?
Start with Bing Webmaster Tools' AI Performance report, then check AI crawler visits in your server logs.
My take
The mainstream reaction will be "finally, we can protect our content, block it." I think the opposite is true for Taiwanese SMBs: the biggest risk here isn't AI learning from you, it's accidentally locking yourself out of AI answers. Publishers live on content, so blocking training makes sense for them; you live on being found, so being trained on and cited is free exposure. I expect a wave of "AI visibility dropped to zero" cases in 2027, caused mostly by plugin and hosting defaults rather than owners' decisions.
For ScriptWalker (a Taiwan Laravel/Flutter custom development studio), our own crawler logs show AI bots mostly fetch English pages, which means what AI can read needs managing. We'll add robots.txt checks, AI crawler logging and question-style service pages to standard website maintenance; maintenance starts at NT$2,000 per month and brand websites at NT$15,000.
- Email: [email protected]
- Phone: 0916-224-047
- LINE: @ufv9089p
Sources
- Google Search Central: Supporting a shared AI training control for websites (2026-10)
- Microsoft Bing: Clearer preferences for training and grounding (2026-10-30)
- Bing Webmaster documentation: New AI preferences
- Reddit Q3 2026 shareholder letter (SEC)
- Search Engine Roundtable: Google adopts new AI training control
- Search Engine Journal: Reddit CEO ties search referral drop to Google AI Mode