Is agent memory better than RAG over the raw conversation?

Many of 11 agent memory systems that have an LLM summarize and restructure conversations ahead of time scored below plain RAG, which keeps the raw conversation and searches it when a question arrives. Across 99 settings (3 benchmarks times 3 models for each system), memory beat two-round RAG in 9 ^[1]. The paper was posted to arXiv on October 25, 2026 by researchers from five institutions, including the University of Illinois Urbana-Champaign and Tsinghua University. It has not yet been peer reviewed, and its authors include Jiawei Han (data mining) and ChengXiang Zhai (information retrieval) ^[1].

This article reads the arXiv paper v1, its text, Tables 1, 2, 4 and 19 and the appendix case figures, as of October 2026, checking each number against the source ^[1]. The paper is also listed on its Hugging Face paper page, and we have not run the code the authors say they will release. The 9 of 99 figure is our own count of the cells in Table 1. We have no relationship with the authors.

Is agent memory better than RAG?

Mostly not, in this paper's experiments. The authors grouped memory systems by how they store information: flat stores (A-Mem, SimpleMem, MemOS), hierarchical or lifecycle designs (MemoryOS, LightMem, Volcengine's OpenViking), graph-based designs (HippoRAG2, PlugMem, MAGMA) and trained policies (Mem-T, Memory-R2) ^[1]. The reference is plain RAG that cuts the raw conversation into chunks, retrieves with BM25 and embeddings together, and reranks. It comes in a one-round version (1rd) and a two-round version (2rd), in which the model judges whether the first results are enough and, if not, issues a follow-up query ^[1].

The authors also matched retrieval budgets. Memory units differ in size across systems, so the same top-k can expose very different amounts of text. They raised each system's top-k until the share of stored tokens shown to the model (Retrieval Scope) was at least that of one-round RAG ^[1].

Table 1. Accuracy and cost summed over 100 LongMemEval-s conversations. Total tokens add build, maintenance and inference tokens (computed from Table 19 of the paper) ^[1].

Figure 1. The best memory system scored 84.0%, six points below two-round RAG at 90.0% ^[1].

LongMemEval gives a model a long history of user and assistant sessions and asks about facts inside it; the s version uses shorter histories. There, two memory systems beat one-round RAG (82.0%): A-Mem and SimpleMem, both at 84.0%. None beat two-round RAG ^[1]. On the other two benchmarks (PersonaMem-v2 and MINT-Eval), a few settings edged past RAG, which is where the 9 of 99 come from.

How much does it cost to build agent memory?

Systems that build memory ahead of time run the whole conversation through an LLM before any question arrives, and the cost shows. In Table 1, A-Mem spent 117.0 hours and about 98,000 LLM calls building memory for 100 conversations. MemoryOS took 126.5 hours and MAGMA 103.2 hours ^[1]. Two-round RAG spent 0.11 hours indexing the same conversations and 7.7 minutes answering, with 0.65M tokens in total ^[1].

Time is not the only cost. Nothing guarantees that the model that builds the memory is the one that later reads it. The authors built memory with one model and answered with a smaller one, and for many memory systems accuracy fell further than for RAG ^[1]. Because both building and reading memory depend on LLM judgment, a model change adds one more place for results to drift.

How do you measure the net value of memory?

The authors split memory's gains and losses query by query. They ran each memory system and one-round RAG on the same questions, and called the questions only memory got right the memory benefit and the questions only RAG got right the memory tax ^[1].

Here $V_{\text{net}}$ is memory's net value, $B$ the share of questions in the benefit set and $T$ the share in the tax set. Bundling scattered facts or distilling preferences so that memory solves a question RAG missed raises $B$. Losing a clue in summarization so that memory fails a question RAG solved easily raises $T$. In many systems $T$ exceeded $B$, so net value was negative ^[1].

The split matters because average accuracy hides it. Two systems with the same accuracy can solve the same questions as RAG or entirely different ones. The authors treat benefit as the intended result of memory design and trace the less studied tax along three axes: faithfulness, accessibility and robustness ^[1].

What is lost when an LLM summarizes a conversation for memory?

Specific numbers and conditions went first. The authors sampled facts from the raw history and traced each into memory units, labeling it no loss, partial loss, major loss or no memory unit. OpenViking never stored 38.0% of the sampled facts and badly damaged 13.0%. Only 39.4% survived intact ^[1].

Figure 2. The share of facts kept without loss ranged from 98.4% for MemoryOS to 39.4% for OpenViking (Table 2) ^[1].

The appendix cases show what the loss looks like. A user said a taxi from the airport to the hotel would cost around $60, which felt pricey. OpenViking stored that as a preference for cost-effective transportation due to budget concerns, so a later question about how much the train would save has no $60 to work with ^[1]. SimpleMem rewrote "my laptop backpack arrived on 1/20" as "started using it on 2023-01-20", changing the evidence for a question about delivery time ^[1].

Deletion requests went wrong too. When a user asked the assistant to forget details about family pressure to marry, SimpleMem stored the request as new facts about the same detail, and PlugMem dropped the request and kept exactly the detail to be forgotten ^[1]. Forgetting turns out to be hard in summary-based memory.

Why can't the system find information it still has?

Facts that survive can still surface late if they are split, diluted or reworded. In one case the user asks for total bike-related spending since the start of the year. The raw turn is a single passage: a chain replacement for $25 and new lights for $40 at an April 20 tune-up. PlugMem split it into three memory units, and the $25 unit needed for the total came back only in the fourth retrieval round ^[1].

To check this more broadly, the authors compared the similarity between a question and its raw evidence turn with the similarity between the question and the matching memory unit. Building memory shifted that similarity distribution, and several systems fully recovered the evidence for fewer questions than RAG in the first round ^[1]. Given gold memory units, MAGMA and LightMem scored close to the gold raw-turn condition. The information was there; it was hard to reach ^[1].

The retrieval bottleneck paper we read earlier showed evidence scattering when raw documents are chunked. This paper shows the same problem happening again at the LLM summarization step.

Are the benchmarks simply easy for RAG?

The authors took the question up directly and agreed that some benchmarks favor retrieval. Running no memory, an IDF keyword baseline that matches only rare words, embedding RAG and hybrid RAG across several long-term memory benchmarks, the bare keyword method passed 50% on some and hybrid RAG reached about 70% ^[1]. When plain retrieval already solves many questions, memory has little room to gain and small losses show up directly in the score.

That does not explain the result away. Many real user questions are retrieval questions too, asking for the exact number mentioned last time. The authors suggest reporting RAG scores alongside memory results so the remaining headroom is explicit ^[1].

They also found results depend on the task. In experiential memory that reuses past task traces, memory that distilled long traces into workflows or strategies sometimes beat RAG on web tasks (WebShop, Mind2Web), while several memory methods trailed RAG on tool use (AppWorld, τ²-Bench) ^[1]. In video world models, reinjecting past frames directly was not consistently beaten by latent or cache-based memory ^[1].

Should agent memory keep the raw source?

Keeping a path back to the source cut the loss sharply. The authors tested two changes that bolt onto existing systems without a new architecture. The first, src-aug, searches the raw history alongside the built memory and reranks both before answering ^[1].

Figure 3. Searching the raw history as well raised accuracy and lowered the memory tax for all three systems (LongMemEval-s, Qwen3.6-27B, Table 4) ^[1].

On LongMemEval-s with Qwen3.6-27B, SimpleMem rose from 73.6% to 82.8% while its memory tax fell from 12.0% to 5.2%. Mem-T went from 75.0% to 80.0% (tax 10.8% to 5.2%), and MemoryOS from 73.4% to 76.4% (tax 11.4% to 7.8%) ^[1]. The second change, recursive, writes past questions and the evidence found for them back into memory; on PersonaMem-v2 it lifted all three systems slightly. It could not be applied to LongMemEval-s, which has one question per conversation ^[1].

The authors also name a deeper tension. Processing ahead of time works without knowing which transformations will help; processing just in time knows the question but slows the response. In personal agents, past interests and access patterns can hint at what will matter, so the authors suggest treating memory partly as a recommendation system ^[1].

How does TRAIL read this result?

We reread TRAIL Search's brand memory against this paper. Brand memory stores the raw references a customer provides (URLs, files and pasted text) and uses an LLM to extract fields such as mood, colors and business category. At generation time the model receives a directive list built from those fields, and no raw excerpt goes with it (code as of October 31, 2026). In the paper's terms, the source is kept but the generation step has no path back to it.

So we added a review item for brand memory: retrieving raw-source excerpts into the generation path. It needs to measure whether the original text fills in figures or wording the extracted fields missed, and how much that adds to generation cost. We have no measured result yet. Our company fact files already keep the raw materials folder untouched and footnote the source of each fact, as described in our post on LLM wikis and the Open Knowledge Format.

Keeping evidence also connects to showing where answers come from. As our posts on RAG source attribution and RAG hallucination defense argue, a wrong answer can be fixed only when its evidence can be traced back to the original text.

Limitations

The study cut its samples for cost. It used 20% of PersonaMem-v2, 10% of MINT-Eval and, for GPT-5.6 Luna, 20% of LongMemEval-s (100 conversations), with stratified sampling by question type ^[1]. The strongest model was GPT-5.6 Luna (low), with no stronger frontier models, and answers were graded by an LLM judge based on Qwen3.6-27B ^[1].

The benchmarks are mostly static, offline and in English. Long-running deployments with interleaved queries and multilingual conversations are not covered ^[1]. Memory systems ran with official settings, and the authors note that further tuning might improve them. Memory tax is defined relative to one-round RAG, so a different reference would change its size ^[1].