How many measurements to trust AI search visibility

AI search visibility cannot be trusted from a single measurement: for the same query, about 65% of the cited sources change within a day, and much of that swing comes from the model generating differently each time rather than from algorithm changes. A St. Gallen team tracked 4 AI search engines for 45 days and found cited sources overlapped on average only 34-42% from one day to the next (Jaccard) ^[1]. So to trust a measurement you repeat it, and this study puts numbers on how many runs and days that takes ^[1].

This article reads the body and tables of the paper posted on arXiv and checks the figures against it. All numbers are the paper's experimental results, and with no license on the repository we have not reproduced them ^[1]. One author also works for a GEO monitoring vendor, and we too sell repeated tracking, so we read the conclusion with that interest in mind. The part about our product is in the last section.

Why does AI search visibility change every time?

Traditional search tends to keep a page in the result set even as its rank moves. Generative search, by contrast, works by inclusion or exclusion ^[1]. A page appears in one answer and is gone from the next ^[1]. The authors attribute this to two things: the probabilistic nature of token generation, and the way RAG compresses many sources into a limited answer space while selecting evidence ^[2].

The team measured this with two datasets ^[1]. One is a time series of 4 campaigns (telecom, real estate, sporting goods, consumer electronics) by 8 prompts by 4 engines, collected once a day for 45 days; the other re-ran the same queries up to 10 times in a single day ^[1]. Overlap was measured with the set-similarity metric Jaccard, where closer to 1 means two citation lists are identical ^[1]. Gemini had many missing collection days and ChatGPT returned many answers with no citation, so both were handled out of the source analysis ^[1].

How much do citations change in a day?

Cited sources overlapped on average only 34-42% from one day to the next (Jaccard, 0.336-0.423 by campaign) ^[1]. By the non-overlapping share, that is about 65% changing in a day ^[1]. Brand mentions were steadier at 45-59%, a figure measured on the 3 campaigns with at least 70% detection ^[1].

The authors read this as a persistent property rather than a one-off anomaly ^[1]. So a single snapshot is hard to call "our brand visibility" on its own ^[1].

Figure 1. Day-to-day citation swing.

Is the cause time, or the model?

The team separated change that accrues over time from the model's own variation ^[1]. Sources re-run at the same moment overlapped by a Jaccard of 0.32-0.43, close to the 0.34-0.42 measured a day apart ^[1]. That a re-run with almost no time elapsed swings this much is an observation that much of the instability comes from the model's probabilistic generation ^[1].

So the noise in any single measurement does not shrink by measuring more often ^[1]. Noise shrinks by averaging, and you average either by repeated runs at one moment or by accumulating several days of observation ^[1]. Prompt-level variation was also large: some queries stayed above a Jaccard of 0.8, while others fell below 0.2 and changed almost every time ^[1].

Figure 2. The cause looks more like the model's randomness than time.

How many runs and days does trust take?

A single measurement has a standard error (SE) of 0.37, effectively no information ^[1]. Even at a true detection rate of 50%, the nominal 95% interval spreads very wide ^[1]. The team treated the 10-run data as the truth and raised the number of runs to estimate the SE ^[1].

Table 1. Standard error and 95% interval half-width of brand-detection rate by number of runs ^[1].

The SE of brand-detection rate fell below 0.10 at 7 runs and below 0.08 at 8 ^[1]. Seeing which URLs are cited converges more slowly, needing 8 runs ^[1]. Because the truth was set as the mean of the same 10 runs, the SE at 7-8 runs may be somewhat optimistic ^[1].

Apart from runs, accumulating one run a day over several days also lowers the SE ^[1]. By window, SE reached 0.08 at 2 weeks and 0.03 at 4 weeks ^[1]. The authors advise at least 7 runs per prompt per day and a 2-4 week rolling window ^[1].

Figure 3. Once is noise; signal starts at 7 runs.

What are the limits of this study?

The authors state clear limits ^[1]. All data was collected from servers in Switzerland and served with a Swiss IP and locale, so it does not generalize to other regions or languages ^[1]. There are 4 campaigns, it is an arXiv preprint with no stated venue, and the code repository has no license ^[1]. Brand detection uses fixed keyword matching, so brands mentioned by a synonym or abbreviation can be missed ^[1].

The authors also note that one of them works for a GEO monitoring vendor ^[1]. A "measure more" conclusion favors sellers of repeated measurement. The same interest applies to us, who also sell repeated tracking, so we kept the conclusion at arm's length.

How does this connect to TRAIL Search?

This study puts numbers behind why we measure visibility repeatedly. TRAIL Search gathers more than ten unbranded discovery prompts and scores them as a median over several runs, attaches a preliminary, medium, or high-confidence badge by the number of runs it measured, and shows the run-to-run flip rate separately. It recommends a two-week to one-month re-measurement cycle. This paper is already on our list of measurement sources.

That said, our default number of runs is kept low for low-cost tracking, so it sits below the paper's high-confidence bar of 7-8 runs ^[1]. We add more runs for decision-grade verdicts. Separately, our threshold for calling a score change noise is not something this paper directly validated, because it measures overlap similarity and standard error rather than point changes in our composite score, so we keep that threshold as a working convention and calibrate it once real re-measurement data accumulates.