AI agents already know when to search the web

LLM agents already know whether they need a tool like web search before they write an answer. A single linear classifier on the hidden state right after reading the question separated tool-necessary tasks with AUROC 0.89 to 0.96, more accurately than the model's own verbalized judgment^[1]. Yet agents call tools whenever tools are available. A simple method that reads this signal and prefills the start of the response cut tool calls by 48% while losing only 1.7 accuracy points (paper abstract)^[1].

The evidence is the WHEN2TOOL paper released by UCSD researchers in May 2026^[1]. They built a benchmark of 18 environments (15 single-hop, 3 multi-hop) with 1,080 training and 2,700 test tasks, and tested six models: Qwen3 (1.7B, 4B, 14B, 32B) and Llama (3.1-8B, 3.3-70B). The authors' released code and benchmark are the primary source here^[2], and we did not reproduce the results ourselves. Figures are taken from the arXiv v2 text as of November 2026.

We build a product that measures how often brands are cited in AI search, so we have an interest in when AI looks things up and when it answers from memory. This paper studies open-model agents, not commercial AI search. We report the figures as written and mark the parts connecting them to GEO as our own reading. Where an earlier study on handing agent decisions to a small model covered what to delegate, this post covers an earlier decision: whether to call a tool at all.

Agents call unnecessary tools by default

LLM agents given tools call them even when the task does not need one. In the default setting, six models made 2,100 to 4,400 tool calls on 2,250 single-hop test tasks, more than one per task (paper section 3.2)^[1]. Even on 750 easy tasks that are mostly solvable without tools, Qwen3-1.7B made 864 calls and Llama-3.3-70B made 1,482. The authors sum up this behavior as "tools are available, therefore use them."

The benchmark splits tool necessity into three categories. Computational scale (12 plus 7 is mental math, trillion-scale multiplication needs a calculator), knowledge boundaries (the capital of France is common knowledge, an obscure historical date needs a search), and execution reliability (a short code output can be traced mentally, deep recursion needs an executor). Each environment has easy, medium, and hard levels, with medium designed as the decision boundary where tool need is ambiguous. The second category, "do I know this?", is the decision to call web search or RAG, and the paper's example is that nobody needs a web search to learn the year humans landed on the Moon.

Prompting fewer tool calls breaks hard tasks first

Prompting the model to use tools sparingly does reduce calls, but it cuts necessary ones indiscriminately. The authors measured the accuracy lost per saved call. On Qwen3-4B-Instruct the cost was -17.3 on easy tasks and -42.4 on hard tasks, so hard tasks lost 2.5 times more accuracy per saved call (paper table 2)^[1]. The prompt takes tools away exactly where they are needed.

Asking the model to reason about necessity before acting (Reason-then-Act) only partly helped. Unnecessary calls fell on easy tasks, but the cost on hard tasks stayed high (-34.7 for Qwen3-14B on hard tasks) and reasoning tokens added overhead. On some models it broke down entirely. Llama-3.1-8B dropped from 79.5% to 31.2% accuracy and Llama-3.3-70B from 83.1% to 47.9%, narrating an intent to call tools without ever producing a valid call (paper section 3.2)^[1]. A handful of instructions also could not hit a chosen point between accuracy and call count. The "only if necessary" instruction behaved almost identically to the default.

The model already records whether it needs to search in its hidden state

Models already encode whether a tool is needed in their pre-generation hidden state. The authors labeled each task by having the model answer without tools, marking it tool-unnecessary if correct and tool-necessary if wrong. They took the hidden state at the last question token across all layers and trained an L2-regularized logistic regression. Trained on 900 examples in seconds on a CPU, the classifier scored AUROC 0.89 to 0.96 on 2,250 test tasks (paper section 4)^[1].

Table 1. AUROC for predicting tool necessity from pre-generation hidden states. Source: Sun et al. 2026, table 3^[1].

The most striking result is Llama. On the same two models where reasoning first collapsed tool calling, the classifier still exceeded AUROC 0.9. The information that a tool is needed is clearly in the representation, but generation fails to act on it. By difficulty, medium tasks scored lowest, matching the design intent of a decision boundary where the model itself is unsure.

Cutting tool call costs by 48%: probe and prefill

Probe&Prefill reduces tool calls by writing the classifier's verdict into the start of the response. If the probe says no tool is needed, it prefills a sentence like "I can solve this directly without using a tool", otherwise "I need to use a tool for this question", and the model continues from there (paper section 5)^[1]. Moving the probe threshold gives continuous control over the point between accuracy and call count, adding under 1ms per task with no extra model calls.

Averaged across six models, it cut tool calls by 48% while losing only 1.7 accuracy points. At comparable accuracy, the best baseline cut only 6% of calls, and the baseline with a similar cut lost 5 times more accuracy (paper abstract and table 4)^[1]. The accuracy cost per saved call was -1.6 on easy tasks and -3.4 on hard tasks, lower than every baseline. Skipping easy calls while keeping hard ones is what separates it from prompting.

Table 2. Accuracy lost per saved tool call (closer to zero is better). Source: Sun et al. 2026, table 4^[1].

On the 3 multi-hop environments, it cut Qwen tool calls by up to 75% while keeping or improving accuracy. The probe also transferred to unseen environments within the same category. Compared with full fine-tuning (SFT), which raised accuracy but did not reliably reduce calls, it was orders of magnitude cheaper (paper appendices C, D, H)^[1].

Search calls fell on a real web search agent too

Outside the controlled benchmark, search calls fell by 20% to 56% in a real search agent setting as well. The authors ran the same experiment with Qwen3-4B-Instruct on six open QA datasets from the Search-o1 framework^[3]. On 4 of the 6, Probe&Prefill cut more searches than the baselines at equal or higher accuracy (paper appendix G)^[1].

Table 3. Accuracy (%) and search calls per Search-o1 dataset, with reduction versus default in parentheses. Source: Sun et al. 2026, table 18^[1].

The abstract's "20% to 56% without accuracy loss" summarizes the good side of this table, so two rows deserve a separate read. On NQ it cut searches by only 6% and accuracy fell 2.8 points. On MuSiQue, which needs 3 to 4 reasoning hops so nearly every question needs a search, the forced-call baseline (-56%) cut more than Probe&Prefill (-48%). Transferring the WHEN2TOOL-trained classifier without retraining lowered AUROC to 0.67 to 0.80 (paper table 19)^[1]. The signal carries over to real QA, but not as cleanly as on the controlled benchmark.

When AI answers from memory, a brand exists only as parametric knowledge

From here on this is our reading. Whether AI search looks things up or answers from memory is already leaning one way before generation, and for questions answered from memory, a brand reaches the answer only through parametric knowledge, without a live citation. We have explained the difference between GEO, SEO, and AEO as live citation versus a share of learned knowledge, and this study shows that the split between the two sits on the model's internal "do I know this?" judgment.

Two implications follow. One is cost pressure. The paper starts from the API fees and latency that unnecessary calls pile up, and as techniques like this spread, agents and AI search will look things up less for questions they judge they already know. Then the more a category question looks like common knowledge, the less room a newly written page has to get in. The other is new products. A search happens only when the model judges it does not know, and the discovery gap, where AI knows the name but does not recommend new products, fits what happens when the model answers from memory alone without searching. Once a search does happen, how AI splits a question into several searches and the queries agents actually send decide exposure.

What this study did not cover

The method carries clear constraints. Probe&Prefill must read hidden states, so it only works on open-weight models and cannot be applied as is to commercial models used through an API. The experiments cover only six models from two families, Qwen3 and Llama, and the real search experiment ran on Qwen3-4B-Instruct alone.

The labeling also has limits. A task is labeled tool-unnecessary if the model answers correctly without tools, a rule that holds only for questions with fixed answers. Cases where the model believes it is right but gives a stale answer, such as facts that changed after training or newly released products, are not captured by this definition. Adding one question of our own: whether commercial AI search decides to search using internal signals like this, or with a separate router or rules, is not something this paper can tell us.

Measure answers with sources apart from answers from memory

In practice, what remains is the unit of measurement. For the same question, an answer where AI search looked up the web and attached sources and an answer from memory without searching give a brand different paths in. If you are missing from the first kind, the work is on your pages and the content that gets cited; if you are missing from the second, the longer work is the mentions and reputation that accumulate in training data.

Our measurement tool asks non-brand questions repeatedly per engine and records how often a brand appears in the answer. After reading this study, we note that those records should be read with responses carrying source links separated from responses without them. Since whether a search happens can itself vary between runs, the first step is to look at that ratio through repeated measurement under the same conditions rather than a single run.