Cutting LLM agent costs: delegate decisions to a 4B model

Most steps an LLM agent takes are multiple-choice decisions, and delegating them to a small decision model under a confidence threshold cut agent costs by 81.1 to 89.0 percent while keeping task success within 3 points, according to the JevAgentHarness paper^[1] from UIUC, Princeton and Google. Two caveats come attached. The decision model most accurate on isolated questions solved only 69.5 percent of real agent tasks, and delegation without a fitted threshold cut success by 19.0 points. Delegation is conditional, not free.

This article is based on the arXiv v1 PDF as of November 2026, read in full and cross-checked against the public model card^[2]. The authors ran 11 registered hypothesis tests with Holm correction, and the success-preservation claims are non-inferiority tests with a 3-point margin (5 points on tau-2-bench). It is a preprint, not yet peer reviewed, and we run multi-step LLM pipelines ourselves, so we have an interest in this topic.

The problem: why agents are slow and expensive

An agent harness calls a large LLM at every step: which tool to invoke, with what arguments, whether the task is done. Long tasks stack these calls up, getting slow and costly. Yet most of these steps are multiple-choice questions over listed options^[1].

A decision model answers such questions without generating text, returning only a probability per option. In one episode the authors recorded, a call to the decision model Jev took 28 percent of an LLM call's time and 2.6 percent of its price^[1]. The framing is Kahneman's: hand decisions to the fast, intuitive System 1 and call the slow, deliberate System 2 only when needed.

In state $s$, the decision model scores every option $o$, and its top answer is used only when that confidence clears the threshold $\tau$; otherwise the LLM decides. The paper's contribution lies in where $\tau$ comes from.

Three-level evaluation: from isolated questions to a running agent

The authors built AgentDecisionBench with three levels^[1].

Table 1. The three levels of AgentDecisionBench. Higher levels let each decision change the state the next one sees^[1].

The environments are tau-2-bench, ALFWorld and their own DecisionGym, and hooks let them test existing harnesses without code changes: mini-SWE-agent and SWE-agent on SWE-bench Verified, Codex CLI and Claude Code (public repository^[3]). The agent LLM is mostly Gemini 3.5 Flash.

The core finding: isolated accuracy does not predict in-loop success

The model with the best L1 accuracy solved only 69.5 percent of L3 tasks^[1]. And delegating tool choices wholesale, without a threshold, dropped task success by 19.0 points.

The reason is that agent decisions are sequential. One wrong choice does not end as one wrong answer; it changes the state every later decision will see. Isolated questions cannot measure that chain. Components that score well in isolation and collapse inside the loop are the same pattern we saw in agent memory paying a tax in practice despite benchmark scores.

The harness: thresholds fitted on the agent's own trajectories

JevAgentHarness runs the decision model in shadow mode first. The agent keeps deciding with its LLM as usual, while the decision model silently answers and records every decision point. Those trajectories are used to fit a separate threshold per decision point, and in production only the decisions that clear their threshold are delegated^[1].

The published fitted values show how conditional this is. Per the model card^[2], the tool-choice threshold is 0.445 while the argument-choice threshold is 0.996: arguments are delegated only under near certainty. And these values assume Gemini 3.5 Flash as the agent LLM, so a different agent needs refitting.

AgentDecision-4B: trained on states reached after mistakes

Existing decision models share one more blind spot: they have never seen the states an agent reaches after its own mistakes. The authors collected states agents actually reach, including post-mistake states, and fine-tuned a 4B model with LoRA (r=32, 2 epochs) using Jev's answers as labels^[1].

Table 2. AgentDecision-4B task success and cost reduction. The savings are specific to these benchmarks and serving conditions^[1][2].

The 4B student beating its teacher by 9.5 points on out-of-distribution tasks is the most counterintuitive result in the paper^[1]. The authors' explanation is the training data: the teacher is strong on general questions, while the student has concentrated exposure to the states near the holes this particular agent actually falls into.

Limitations

• Benign states only. Adversarial inputs are out of scope, and the authors note that cheaper agent control also lowers the cost of misuse.
• English tasks only, and the agent LLMs are mostly Gemini models.
• Thresholds and training are both fitted to a specific agent's runs; a new agent, new tools or a new task family require refitting and retraining.
• For coding agents, whose actions are free text, multiple-choice delegation does not apply; the decision model could only route which LLM writes a step, saving 7.0 to 10.5 percent.
• No comparison against learned routers such as RouteLLM on the same tasks, and prices and latencies are as recorded in the authors' environment.

What it means: a third way to run AI agents on small models

After carving a small LLM out of a big one with Sheared-LLaMA and owning a domain model on open weights, this paper shows a third path to owning a small model: delegate only the decisions, leave the rest of the agent unchanged, and the key ingredient is your agent's own execution traces rather than a giant pretraining corpus.

From a measurement point of view, the evaluation design deserves the second look. The top isolated-accuracy model landing at 69.5 percent in the loop^[1] means component scores and system outcomes diverge, sometimes to opposite conclusions. It is the same principle as retrieval-pool entry and final citations splitting in AI search: in any domain, the only number you can trust is the one measured inside your own loop.

You can continue with MMDocRAG: can AI pick evidence in multimodal document RAG?.