Are AI Visibility Tools Accurate About ChatGPT Citations? A Paper Finds the API Plus Google Matches Only 26–28% of Cited Pages, and 40%+ of Pages Appear Once in 20 Runs: SMBs Should Track Site-Level Presence

Short answer: Not entirely. A paper released on Oct. 11 found that using the OpenAI API plus Google results as a stand-in for ChatGPT recovered only about 26–28% of cited pages, and over 40% of pages showed up in just one of 20 runs. SMBs should track site-level presence, not single-page rankings.

On Oct. 18, Search Engine Journal covered a paper on arXiv by researchers from the National University of Singapore and Xi'an Jiaotong University. They ran 80 real questions 20 times each in a signed-in ChatGPT web app, recorded which pages were actually cited, and compared them with the OpenAI Responses API and the first five pages of Google results. The number that stops you: the two proxies combined matched only 28.3% (GPT-5.3) and 26.2% (GPT-5.5) of cited pages. In one sentence, most AI visibility reports measure something other than the citation list users actually see.

Why does this matter now? When OpenAI announced GPT-6 on Oct. 7, it said more than 1.2 billion people use ChatGPT each week, and "being cited by AI" has become a new marketing KPI. Over the past 18 months, tools that track AI answers have expanded fast. Category leader Profound raised a $96 million Series C at a $1 billion valuation in February, bringing total funding to $155 million with more than 700 enterprise customers. But to collect data at scale and reliably, the easiest route is to call the official API or scrape Google, and the paper shows both shortcuts miss. Model changes also reshuffle citations: SISTRIX saw day-to-day citation variation in German answers jump from the usual 1–2% to 47% when GPT-5.5 rolled out, and GPT-6 is only two weeks old.

The tool category has already split. Profound starts at $99 a month, billed annually; Semrush AI Visibility costs $99 per domain per month; Otterly.AI starts at $29. On the other side, Germany's Peec AI has raised about $29 million and serves more than 2,500 customers, according to Sacra, and uses browser automation to simulate real users, arguing that API answers differ from what users see. The category is still raising money, but "where the data comes from" will decide the next shakeout.

For Taiwan's SMBs, the impact is more direct: nearly 70% of Chinese prompts searched only in English, so shops with Chinese-only websites may never enter the candidate pool. Below we break down the numbers, then give actions for owners, marketers and developers, plus a sampling method that needs no subscription.

Event details: the gap between the API, Google and ChatGPT's actual citations

Bottom line: the proxies match about half at the site level, and only a quarter at the single-page level.

• Study design: 80 questions from the public Search Arena dataset, 40 in English and 40 in Chinese, covering recommendations, fact lookup, information synthesis and analysis; data was collected mainly between March 29 and July 29, 2026.
• Google alone: matched only 9.4% (GPT-5.3) and 6.4% (GPT-5.5) of cited pages, and 26.1% and 22.8% at the site level.
• API alone: the API's full source list matched about 22.5%–22.9% of pages; counting only pages the API actually cited, the match rate was 16.4% to 23.3%.
• Combined: 26.2%–28.3% of pages and 48.7%–50.3% of sites.
• Query rewriting: in GPT-5.3 and GPT-5.5 runs with visible searches, ChatGPT never searched with the user's exact wording; the OpenAI help page also says ChatGPT rewrites questions into one or more queries.
• Every run differs: across 20 runs of the same question, 44.5% of pages cited by GPT-5.5 appeared only once.

Winners are tools that can prove their data comes from the real interface, and brands willing to sample for themselves. Losers are teams making decisions from single-page ranking reports.

What three kinds of readers should do now

Bottom line: owners change how they read reports, marketers add English pages, and developers automate sampling.

• Brand owners: ask your agency or tool to report "our domain's presence rate across 10 questions," not where one page ranks. A single screenshot is not proof of results.
• Marketing and SEO practitioners: create English versions of 3–5 key service pages that state your area, price range and specs clearly, so you have a chance when Chinese prompts become English searches.
• Developers and agencies: use the web search tool in the Responses API with the sources field enabled to batch-log the domains the model consulted as a "candidate pool" metric, and label clearly in reports that it is not ChatGPT's actual citation list.

AI visibility tool comparison

Bottom line: prices differ by more than 3x, but first ask whether the data comes from an API or the real interface.

Prices compiled from Ryze's 2026 comparison; check each vendor's checkout page for current pricing.

What they won't tell you

Bottom line: the paper has its own limits and does not invalidate every tool.

• Counterpoint 1: the real interface is unstable too. Even with browser simulation, 40% of pages appeared once in 20 runs, so too few runs still mislead, and the extra cost lands in your subscription.
• Counterpoint 2: limited sample and versions. Only 80 questions, GPT-5.2 to 5.5, and a preprint that has not been peer reviewed; GPT-6 numbers may look completely different.
• The overlooked long tail: in two collection sessions the system switched models after 14 and 16 runs. If a tool doesn't log model versions, its trend chart may mix two models.

A no-subscription alternative for SMBs

Bottom line: 10 questions, 5 runs each, domains only; one afternoon a month beats a single screenshot.

• Step 1, pick questions: Choose 10 buying questions customers often ask, such as "best ___ in Taichung".
• Step 2, repeat: Ask each one 5 times in new chats in the ChatGPT web app.
• Step 3, log domains: Log cited domains in a spreadsheet; full URLs are not needed.
• Step 4, compute presence: Calculate your domain's presence rate monthly, plus an extra check after model updates.

• ☐ Note the model name used for every record
• ☐ Prepare an English version of each Chinese question for comparison

FAQ

Can I still trust AI visibility tool numbers?

Use them for trends, not page rankings. The API and Google together matched only 26–28% of cited pages; site-level matches near 50% are more reliable.

If I ask ChatGPT in Chinese, does it search in Chinese?

Mostly not. About 67%–70% of Chinese prompts with visible searches used only English queries, so Chinese-only sites are easily skipped.

How many times should I repeat a prompt when tracking?

At least five times per question. Over 40% of pages appeared in only one of 20 runs, so a single run is largely chance.

Does the study still apply now that ChatGPT runs GPT-6?

The direction holds, but the numbers need retesting. The paper covered GPT-5.2 to 5.5, and the authors say monitoring must be re-evaluated after model changes.

My take

The mainstream view is that buying an AI visibility tool solves GEO. I think the opposite: single-page AI ranking reports are false precision. Within 12–18 months, these tools will be pushed from selling "where you rank" to selling "the share of answers your site appears in," and prices will come down. The reason is simple: if researchers running 20 repeats can't pin down single pages, a $99-a-month plan that asks each question once can't either.

For Taiwan's SMBs, the better investment is being found by English searches. ScriptWalker (a Taiwan-based custom Laravel/Flutter development studio) sees a service opportunity for agencies here: an "AI citation sampling plus English key pages" package with a monthly sampling report, 3–5 English service pages and structured data. Based on ScriptWalker's experience, setup runs roughly NT$30,000–80,000, followed by monthly tracking bundled with site maintenance. For a free initial sampling assessment, contact us:

• Email: [email protected]
• Phone: 0916-224-047
• LINE: @ufv9089p

Sources

• arXiv: Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations (2026-10-11) (primary research)
• OpenAI Help Center: Searching the web with ChatGPT (primary)
• OpenAI API docs: Web search tool (primary)
• OpenAI: GPT-6 for everyone (2026-10-07) (primary)
• OpenAI Help Center: GPT-6 and other models in ChatGPT (primary)
• Search Engine Journal: OpenAI's API Only Partly Matched ChatGPT's Citations, Paper Finds (2026-10-18)
• Search Engine Journal: ChatGPT citations changed after GPT-5.5, SISTRIX data shows
• Fortune: Profound raises $96 million (2026-02-24)
• Sacra: Peec AI
• Ryze: AI visibility tools pricing compared 2026