How AI product recommendations vary across tools and APIs

Ask AI search "which phone has the best camera?" and ChatGPT names one pick in the first person in roughly four of five answers, while Google AI Overviews does so in 2%. Submit the same question to the API and its sources overlap the consumer screen by only 12.0% ^[1]. This study is a September 2026 arXiv preprint from researchers at Maastricht University's Law & Tech Lab and the universities of Utrecht and Zurich. It has not been peer reviewed, and the authors approach it from a consumer-protection and regulatory angle, with the EU's Digital Services Act and Unfair Commercial Practices Directive in view ^[1].

This article reads the arXiv paper v1, its text, Table 1, and Figures 1 to 7, as of October 2026, checking each number against the source ^[1]. We did not run the data or code the authors say they released. We have no relationship with the authors.

How decisively do AI chatbots recommend products?

AI-visibility research has mostly measured which brands a model names. This study goes one level deeper, comparing how differently each tool frames its answer to the same product-recommendation query ^[1].

The material is ConsumerQ: a dataset of 2,528 commercial-advice queries drawn from real user conversations with LLMs (LMSYS-Chat-1M and WildChat-1M). From these, 117 general physical-product queries without geographic detail were selected (median 8 words) for the audit ^[1].

Of 836 responses extracted from the interfaces, 700 recommended at least one product or brand, and the framing was measured on those. The starkest difference was first-person voice: a product was recommended in the first person, like "my pick", in 79% of ChatGPT answers, 7% of Gemini answers, and 2% of AI Overviews ^[1].

A product was labelled "best" in 73% of ChatGPT answers, 43% of Gemini answers, and 48% of AI Overviews. Conversely, Gemini (94%) and AI Overviews (97%) asked a clarifying question more often than ChatGPT (82%). Products were grouped by category in 71% of ChatGPT answers, 65% of Gemini answers, and 44% of AI Overviews ^[1].

Figure 1. The same commercial question arrived in a different shape depending on the tool. ChatGPT picked one and called it "my pick", while AI Overviews leaned toward listing options ^[1].

Do AI product recommendations change when you ask again?

The authors measured how often the three repeats of a query disagreed. Grouping products by category (26–37%) and calling a product "best" (27–40%) wavered fairly often regardless of tool, while whether any product or brand was recommended changed in only 11–13% of queries, so the decision to recommend at all was relatively stable ^[1].

The products themselves moved more. The Jaccard overlap of recommended-product sets across three repeats was highest for AI Overviews (0.421), then Gemini (0.287), then ChatGPT (0.178). ChatGPT's between-repeat overlap was statistically indistinguishable from its overlap with AI Overviews, so asking the same tool again differed as much as switching tools ^[1].

Because product names are inconsistent ("Apple iPhone 14" and "iPhone 14" count as different products), these overlaps are lower bounds. Even so, a single response cannot characterise what a tool recommends. We cover why a single read is not enough in AI visibility read once is not reliable ^[1].

Do AI tools show different sources for the same question?

The authors compared displayed sources at the domain and page levels. At least one source appeared in 78.6% of ChatGPT and 79.0% of Gemini responses, and among responses with sources the median was 3 domains for ChatGPT and Gemini and 5 for AI Overviews ^[1].

Displayed sources differed sharply by tool. For the same query and pass, ChatGPT and Gemini shared an average of 5.4% of domains, and 76.7% of comparisons shared none. ChatGPT and AI Overviews shared 5.2%, Gemini and AI Overviews 9.8%, and page-level overlap fell as low as 2.4–6.5% ^[1].

Figure 2. For the same question, two tools shared only a single-digit percentage of source domains. Repetition kept surfacing new sources ^[1].

Repetition kept adding new sources. Domain overlap between repeats was 26.0% for ChatGPT, 29.8% for Gemini, and 45.9% for AI Overviews, and ChatGPT's domains per query grew from 2.65 after one request to 5.56 after three. The most frequent domains differed too: techradar and tomsguide for ChatGPT, pcmag, reddit, and youtube for Gemini, reddit and youtube for AI Overviews. Why source tendencies differ by tool is discussed in why a single AI-visibility score is risky ^[1].

The mix of source types also differed. Editorial and product-review outlets made up 56.7% of displayed domains for ChatGPT, 45.2% for Gemini, and 25.4% for AI Overviews, and AI Overviews combined three or more source types in 61.7% of answers, far more than the roughly 19% for the other two ^[1].

Does measuring AI visibility by API match the consumer screen?

Audits usually run through provider APIs, which fix the model and request conditions for easy control. The authors submitted the same query to the interface and the API almost simultaneously (median delay 0.17 seconds) to test whether the API reproduces the consumer screen ^[1].

It did not. Displayed source domains overlapped only 12.0% for ChatGPT and 14.8% for Gemini between interface and API. No domain was shared at all in 60.9% of ChatGPT pairs and 43.4% of Gemini pairs, and most page mismatch came from entirely different domains entering (79.8% and 82.5% of the union) ^[1].

Figure 3. API-measured sources barely overlapped the consumer screen. The pages search returned and the pages shown in citations also differed widely ^[1].

The visible source layer differed too. In the ChatGPT API, search returned a mean of 37.38 pages per query, but only 3.22 appeared in the final citations, so only 27.5% of returned domains and 9.5% of returned pages made it into the citations. How retrieval differs from citation is covered in how retrieval and citation differ ^[1].

This comparison is not an effect of access method alone. Interface and API also differ in model configuration and other provider-controlled components, so the authors measured whether the two conditions reproduce one another rather than attributing the difference to access method. Which method to measure with is covered in API or the actual screen ^[1].

What are the limits of this study?

The authors' own limits first. The audit covers 117 general physical-product queries without geographic detail, so it does not represent all commercial advice. The queries are mostly English while responses were collected in the Netherlands, and the systems were captured at one point in time. AI Overviews content was recoverable in only 134 of 351 searches, so those results apply only to searches where an overview appeared ^[1].

Other widely used assistants such as Claude were not covered, and the interface–API comparison does not establish why the two differ. Whether Korean queries or Korean AI search show the same pattern needs its own measurement ^[1].

What this means for TRAIL

TRAIL Search faces this question directly. We measure ChatGPT, Gemini, Perplexity, and Claude through provider APIs, and Google AI Overviews and Naver's AI briefing by reading the consumer-facing overview block itself. As this paper argues, API-measured sources from chat engines are a different layer from the logged-out consumer screen, so a report should state which layer each number comes from.

The point about repetition applies too. TRAIL Search can re-run the same query multiple times and on a schedule, but the default is a single pass. As this study shows, one response captures only part of the source set, so for commerce queries where products and sources shift often, raising the number of repeats brings the measurement closer to honest. Marking answers whose sources could not be retrieved as not measurable, rather than as a zero mention, belongs to the same stance (how to tell no mention from not measurable).

It lands on commerce too. "Did AI recommend us?" is not a single yes or no; it needs a per-tool, repeated read. Where yesterday's how often AI answers name people was about what a counting instrument misses, this paper is about how the screen and the layer you measure change the result.

Lucas G. Uberti-Bona Marin et al., "'If I Had to Buy Just ONE: Galaxy S26 Ultra': Auditing AI-Generated Product Recommendations", arXiv 2609.18729 (2026).