Does AI search use your exact question? 70,000 queries show

In AI search, people do not write the queries that go to the web. All 69,721 queries in Perplexity's production sample were written by an agent: 17.7% were primary queries restating the user's intent, and 82.3% were support queries the agent added ^[1]. The numbers come from v2 of the Q2D-Web paper, posted to arXiv by Perplexity researchers in September 2026. It has not been peer reviewed, and the company's own embedding model ranks first on the benchmark's primary metric, which is worth keeping in mind ^[1].

This article reads the arXiv paper v2, its text, Tables 1 to 3 and Figures 1 to 9, as of November 2026, checking the numbers against the source ^[1]. The data is private, so we did not reproduce anything, and we only confirmed that the public Q2D-Web leaderboard exists. We have no relationship with the authors.

Does AI search use the user's exact question?

Not in this sample. The agent writes each query from the user's message, the earlier conversation and results already retrieved, and users do not enter queries directly. The authors split a search into one primary query that restates the intent and several support queries that rephrase it or reach for background material and adjacent entities ^[1].

After filtering, 12,365 primary and 57,356 support queries remain, about four support queries per primary. Queries span ten languages: English is 65.8% and Korean 2.2% ^[1].

Figure 1. One user message splits into a primary query and support queries, which include model names the user never typed ^[1].

The paper's example shows the gap. Asked to rank the Samsung A series by price and performance, the agent adds support queries such as "Samsung A56 A36 A26 A16 A06 specs prices", with model names the user never mentioned ^[1]. Wording matters too. An ECIR 2022 study the authors cite reports that even intent-preserving reformulations lower pipeline nDCG@10 by about 20% on average ^[2]. That figure is not a Q2D-Web measurement.

How were the retrievers evaluated?

The authors tested BM25, ten dense embedding models and two late-interaction models on about 190M documents. The corpus is the deduplicated union of the top 5,000 production results for each query, so it holds more plausible wrong answers than a random web sample ^[1]. A first-stage retriever exists to gather candidates for later stages, so the primary metric is Recall@1000 ^[1].

Here $R_q$ is the set of relevant documents for query $q$ and $T_k(q)$ is the top $k$ documents a retriever returns. A higher value means later stages can see more evidence. The authors note that the first-stage retriever bounds what the agent can read and cite ^[1].

Table 1. Five of the 13 retrievers on the full corpus (Table 3 and §5.1 of the paper) ^[1].

The leader changes outside the primary metric. On Recall@100 and nDCG@10 under combined labels, Nemotron-3-Embed-8B came first ^[1]. Which model is best depends on which metric you read.

Are documents an AI did not cite irrelevant?

No. The authors judged relevance three separate ways: documents the agent cited, top production-ranking results (43.1 per query on average), and a combined set that adds LLM judgments (88.5 per query on average) ^[1].

Their explanation matters for anyone measuring citations. The agent stops citing once a claim is supported, so a relevant document that repeats an already cited source stays uncited and gets labeled non-relevant. Citation labels are therefore high-precision and low-recall by construction ^[1]. A citation count can signal that a page was picked. Zero citations cannot be read as irrelevant.

The relative order of retrievers stayed largely stable across the three sets ^[1]. For choosing a ranking, the label source mattered less than one might fear. Absolute values differ widely, though: the same model scores 85.03 on citation labels and 65.73 on web-ranking labels.

Is BM25 keyword search still needed?

This paper alone cannot settle it. BM25 had the lowest Recall@1000 of the 13 (45.25) but recovered 36,073 relevant documents no other retriever found, the most of any model and about 1.3 times the runner-up, pplx-embed-v1-4b ^[1]. BM25 was also the only retriever to improve on support queries (43.3% to 45.7%), while the neural models fell by 0.5 to 3.2 points ^[1].

Figure 2. BM25 had the lowest average recall and the most unique relevant documents ^[1].

Two caveats apply before taking this at face value. Of the nine retrievers that built the judging pool, BM25 is the only one that was also evaluated, and the web-ranking labels come from an internal system using BM25 and dense retrieval together ^[1]. BM25's finds may have had more chances to be labeled relevant, which is our reading. Scale matters as well. Multiplying queries by judgments per query gives roughly 6.17M combined judgments, so 36,073 is under 1% of them (our calculation).

The authors did not test whether adding BM25 to an ensemble raises recall; they propose it as future work ^[1]. It connects to our earlier post on query fan-out and diversity rewards, but this paper does not answer it.

What documents does embedding search miss?

All 13 retrievers left 298,868 relevant documents outside their top 1,000, 6.3% of the relevant set. When the authors classified a sample of 4,000 with DeepSeek-V4-Flash, documents sharing almost no words with the query were the largest group at 28.3% ^[1].

Figure 3. The largest reason everyone missed a document was a vocabulary gap with the query. The classification is an LLM judgment ^[1].

Table 2. Classification from §5.2 of the paper. The classifier is the same model that produced the LLM labels ^[1].

The 18.2% needs context. For cost reasons every retriever read only the first 512 tokens, while the average document runs about 3,300 tokens ^[1]. How real services handle this is outside the paper's scope. Putting the key point early still reduces the risk of this kind of truncation, which is our reading.

Errors run the other way too. Among documents all 13 ranked in their top 1,000 but labeled non-relevant, 35.6% covered a different aspect of the same topic, 19.6% were only partial, and 14.5% concerned a similar but different product, person or version. Another 16.8% were re-judged as actually relevant ^[1].

How do results differ on Korean queries?

On Korean queries the gap between BM25 and the top model was half that on English. Reading Recall@1000 from Figure 7, English shows 71% for pplx-embed-v1-4b and 45% for BM25, a 26-point gap, while Korean shows 68% and 55%, a 13-point gap ^[1].

Table 3. Four languages and four models taken from Figure 7 of the paper ^[1].

The authors do not analyze why. Korean is 2.2% of queries and 3.0% of the corpus, and effects from text analysis or corpus makeup are not separated, so this does not show keyword search works better in Korean ^[1]. Topics vary as well: pplx-embed-v1-4b reached 74% on history and food but 66% on local and 61% on news ^[1].

How does TRAIL read this result?

TRAIL Search can store the queries an engine's response returns and export them to CSV (code as of November 2, 2026). The fan-out in our keyword domain is generated by us, so it is stored separately. We have no metric comparing the two yet, and nothing shows that API-reported queries match everything a consumer app searches internally. So we added a review item: how much of the queries observed from engines our generated fan-out covers. We have no measured result yet.

This connects to our post on the gap between tracked prompts and fan-out. Page diagnosis is also under review. Our chunk simulator can score against question terms, and if vocabulary is the largest reason for a miss, scores against agent queries belong beside scores against the user's sentence. Measurement noise is covered in our post on measuring repeatedly.

Limitations

The queries, corpus and judgments are private, so outsiders cannot reproduce them. The authors say this prevents training contamination, and outside evaluation means submitting open-weight models to the leaderboard ^[1]. The queries reflect one commercial service, and nothing supports generalizing its support-query share to other engines. Filtering works per query, so some searches lost their primary query ^[1].

Citation and ranking labels come from Perplexity's own systems, and LLM judgments rest on a single model, DeepSeek-V4-Flash. The authors acknowledge each label source's bias, and 16.8% of the non-relevant sample was re-judged ^[1]. The company's own model was evaluated and came first, and most follow-up analysis covers only combined labels and Recall@1000. The paper evaluates first-stage retrieval, so it does not directly answer how to write content that gets cited ^[1].