Google Throttled the SERP Data Tap to 20%: Nozzle's Scrape Volume Fell 80% and Your Rank Report Is Now a Poll

At 7:55 a.m. ET on September 18, 2026, Barry Schwartz of Search Engine Roundtable published a story with no press conference and no vendor statement behind it: since roughly September 13, Google has become markedly more successful at blocking third-party scrapers. Derek Perkins, founder of rank tracker Nozzle, supplied the number that matters — the volume of data his tools could retrieve from Google Search dropped about 80%, and switching to DataForSEO as a source produced the same collapse. That same day, the Sistrix status page posted a September 16 notice conceding that its data collection is now running at a reduced rate.

This is not an isolated incident. It is a year-long curve that finally went vertical. In September 2025 Google quietly retired the &num=100 parameter, instantly turning one request for the top 100 results into ten. A Search Engine Land analysis of 319 properties found that 77% of sites lost keyword visibility and 87.7% lost impressions in Search Console — those were not ranking drops, that was bot traffic being drained out of the numbers. Since then: mandatory JavaScript rendering, goto redirects, sign-in checks, and now human-verification gates, layered one on top of another. The market paying for this data is not small. Semrush alone closed 2025 at roughly $471 million in ARR, and Grand View Research puts the SEO software category on a 12.9% compound annual growth path. That is a billion-dollar industry built on somebody else's server choosing to answer.

The blocking is also no longer Google-only. On the same day Schwartz documented Bing showing human-verification challenges to some searchers — evidence that anti-automation is a shared posture across search engines rather than one company's temporary policy. Meanwhile, the AEO/GEO category is at peak capital: Profound closed a $180 million Series D at a $1.8 billion valuation on September 15, just seven months after its Series C. But the way that entire cohort measures "how often a brand shows up in AI answers" is structurally identical to rank tracking: high-volume, patterned, non-human simulated queries. What Google just demonstrated is that this acquisition channel can be closed at will.

For small and mid-sized businesses, the damage is not "will the vendors fix it." It is that your last two weeks of reporting may be lying to you and you have no way to tell. The rest of this article covers three things: how to detect contaminated data, which metrics still hold, and how to move your measurement stack in-house for less than the price of a coffee per month.

What Actually Got Cut

The 80% collapse Perkins describes is Nozzle's retrieval success rate against Google result pages, dated to around September 13 — sitting precisely between the September 4 and September 12 ranking reversals Schwartz logged and the September 15 update. That overlap is itself the problem: when blocking and algorithmic volatility happen simultaneously, the "violent ranking swings" on your dashboard may be an artifact of failed sampling rather than a real movement.

Glenn Gabe's comparison on X made it concrete. He had a site he knew with certainty dropped to near-zero in the 9/4 update and then fully recovered — yet the three major tools drew three mutually inconsistent curves for that window. One fact, three versions.

Who loses: every team treating third-party rank data as a KPI, especially agencies and freelancers shipping monthly client reports; and the vendors themselves, squeezed between a 10x request multiplier, rising proxy-pool costs, and verification gates. Sistrix's phrasing was honest — data is flowing, just slower. Who wins: Google, whose SERP corpus is its moat, and every measurement method that was already first-party — Search Console, GA4, server logs. Google cannot block those, because they are your own property.

Immediate Actions for Three Audiences

Brand owners and SMB operators

• Do not make budget decisions off this month's SEO report. Ask your vendor to annotate data-collection completeness; an inability to produce it means nobody is monitoring it.
• Move the KPI from "average position" to Search Console clicks and conversions. Rank is a rented metric. Clicks are yours.
• If you buy AI visibility monitoring, ask one question: is the data from official API calls or from simulated user scraping? The answer predicts its shelf life.

Marketers and SEO practitioners

• Add a data-quality header to every report: collection window, missing days, and whether it spans September 13.
• Flag all pre-September-2025 impression year-over-year comparisons as non-comparable — the num=100 removal already reset that baseline once.
• Work in relative change within Search Console (cohorted periods, page groups) rather than absolute values, to blunt baseline drift.
• Establish a single source of truth: every externally reported traffic and impression figure comes from GSC/GA4 only; third-party tools are for competitive direction, nothing else.

Developers and agencies

• Pull daily data via Search Console API's searchAnalytics.query — official, free, unblockable, and it preserves the 16-month history window.
• Turn on GA4 BigQuery Export so event-level data lands in tables you own; switching analytics vendors later costs nothing.
• Parse server logs and segment GPTBot, PerplexityBot, ClaudeBot and Google-Extended by User-Agent — the only leading indicator of AI exposure that does not rely on guessing.
• Package those three pipelines as a fixed deliverable. It is far more policy-resilient than reselling seats.

SaaS Comparison

The point is not price. It is the last row: the only option untouched by this event is the one that does not have to scrape anybody.

What Nobody Will Tell You

• Counterpoint one: "Google swept AI crawlers and hit SEO tools by accident" gets the causality backwards. Large AI vendors mostly hold licences or commercial agreements. The traffic that is genuinely high-frequency, patterned, non-human and unpaid is rank tracking. SEO tools are not collateral damage — they are the target.
• Counterpoint two: vendors will fix it, but the definition of "fixed" will quietly change. With 10x requests, rising proxy costs and verification gates, the rational engineering trade-off is to raise prices or to drop refresh cadence from daily to every three-to-seven days and model-fill the gaps. You think you are buying live data; you will receive smoothed estimates, with no error bar disclosed.
• Counterpoint three: Search Console is not a clean control either. Removing num=100 drained bot-inflated impressions, which means the baseline was reset in September 2025. Every year-over-year comparison spanning that point is distorted, and almost nobody annotates it.

The Zero-Subscription SMB Alternative

A build path that can run under US$5 a month:

• Week 1: Enable the Search Console API. Write a daily cron calling searchAnalytics.query with query, page, device and date dimensions, landing into SQLite or the BigQuery free tier.
• Week 2: Connect GA4 to BigQuery Export (daily batch is free) and align sessions and conversion events to the same date table.
• Week 3: Pull access logs from your host, regex-match AI crawler User-Agents, and chart daily pages crawled plus 404 ratio.
• Week 4: Attach Looker Studio (free) and build three views: click trend, page-group performance, AI crawler volume.
• AI visibility: No subscription required. Fix 30 core prompts across four platforms (ChatGPT, Perplexity, Gemini, Copilot), run them manually once a week, and log mentions and cited sources in a spreadsheet. A small honest sample beats a large sample of unknown provenance.

FAQ

My rank tool numbers dropped — is that a Google penalty or a collection failure?

Check Search Console first. If GSC clicks and impressions did not fall in step and only third-party tools show a collapse, it is almost certainly collection. Treat it as a real ranking change only when the divergence between the tool and GSC persists beyond two weeks.

Do I still need a paid rank tracker?

Yes, but demote it. Paid tools remain the only way to see where competitors rank — Search Console only gives you your own data. The sane split: your performance from GSC, competitive direction from paid tools, and never the two in the same KPI table.

Will AEO/GEO visibility tools hit the same wall?

Structurally, yes. Most rely on large volumes of simulated prompts to harvest AI answers, which is the same behaviour class as SERP scraping. The only difference is that OpenAI and Perplexity are still buying market share and are therefore tolerant. Once monetisation tightens, the same verification gates appear — Bing has already started.

What is the single highest-value thing an SMB can do right now?

Back up raw Search Console and GA4 data to your own database every day. GSC retains only 16 months, and every future argument about whether traffic "really" declined will be settled by whether you hold a complete long-term baseline. Starting today versus next year is the difference of one unrecoverable year.

My Take

The consensus reaction is "vendors will route around it, give it a few weeks." My read is the opposite: they will route around it, but the cost will formally degrade third-party SERP and AI visibility data from a subscription data source into an opinion poll, well before mid-2027 — and the degradation will not be announced. It will show up as quietly slower refresh cadences and increasingly vague methodology pages. The question to press is not "have you fixed it" but "what is your sample coverage and what is your refresh lag." Not one vendor surfaces those two numbers on a dashboard today, which is itself the answer.

The less intuitive layer: this is not bearish for the AEO category that just absorbed $180 million. It is the moment its valuation logic splits. The survivors will not be whoever scrapes the most; they will be whoever secures official platform data licences, or pivots to instrumenting clients' first-party telemetry. Scraping-as-a-service becomes low-margin labour within 18 months.

For an agency like ScriptWalker the opening is concrete: productise first-party measurement infrastructure — Search Console API ingestion, GA4 BigQuery Export, AI crawler log parsing, and a Looker Studio dashboard — as a one-time build plus monthly maintenance. What the client buys is not tooling; it is the guarantee that their reporting does not break when somebody else changes a policy. That pitch was hard to make before September 2026. It now has a reference case.

• ☐ Confirmed whether this month's SEO report spans the 9/13 collection break
• ☐ KPI moved from third-party average position to GSC clicks
• ☐ Daily Search Console API ingestion is live
• ☐ GA4 BigQuery Export is enabled
• ☐ Server logs can identify AI crawler User-Agents
• ☐ Pre-September-2025 impression YoY comparisons flagged non-comparable

Need Help

• Email: [email protected]
• Phone: 0916-224-047
• LINE: @ufv9089p

Sources

• Primary | Search Engine Roundtable: Google May Be More Successful In Blocking Scrapers & Tracking Tools (2026-09-18, with direct responses from Nozzle and Sistrix)
• Primary | Sistrix official status page (2026-09-16 reduced-rate notice)
• Primary | Google Search Console API: searchAnalytics.query documentation
• Primary | GA4 BigQuery Export documentation
• Primary | Search Engine Roundtable: Bing Also Asking To Verify You Are Human (2026-09-18)
• Third-party | Search Engine Land: 77% of sites lost keyword visibility after Google removed num=100
• Third-party | Search Engine Journal: Google Causes Global SEO Tool Outages
• Third-party | TechCrunch: Profound hits unicorn valuation with $180M Series D
• Third-party | DataForSEO SERP API pricing