Why RAG misses the evidence an answer needs

RAG usually retrieves one piece of the evidence a question needs, but rarely all of it when several are required. Across 2,255 news-article questions, at least one required fact appeared in the top 10 chunks for 82% of questions, yet all the required facts appeared for only 19% ^[1]. The study is an October 2026 arXiv paper by researchers at MBZUAI and Inception42 in the UAE. It has not yet been peer reviewed, and the authors say their goal is to make RAG's structural limits measurable, not to replace RAG with long-context models ^[1].

This article reads the arXiv paper v1, its text, Tables 1, 3, 4 and 5 and Figures 1 to 3, as of October 2026, checking each number against the source ^[1]. We did not run the code or data the authors released. We have no relationship with the authors.

Does RAG retrieve all the evidence a question needs?

RAG systems are usually judged only on whether the answer is right. That mixes two failures: the retriever not bringing back the needed passages, and the model not using the passages it got. To separate them, the authors defined evidence-based generation (EBG). The model must return the answer together with the input line numbers that support it, and an answer counts as correct (Ans+Evid) only when both the answer (Ans) and the evidence (Evid) are right ^[1].

They started with retrieval. MultiHop-RAG is a benchmark of questions over 609 news articles, each needing 2–4 pieces of evidence spread across different articles. The authors cut the articles into 256-token windows, giving 6,296 chunks, and ranked them by similarity to the question with OpenAI's text-embedding-3-large ^[1].

The result was a gap between any and all. At least one required fact appeared in the top 10 chunks for 82% of questions, but all 2–4 facts appeared for only 19%, and for 8% of inference (multi-hop) questions. Widening to the top 50 brought all-facts coverage to 57%, and the top 100 to 71% ^[1].

Figure 1. Finding one required fact was easy; finding all of them was hard. In the top 10 chunks the two rates were 82% and 19% ^[1].

The authors read all-facts coverage as the ceiling on how faithfully RAG can answer. If even one required fact is missing, the answer cannot be faithful to the evidence, so in the top 10 more than 90% of inference questions are already lost before the model is even called ^[1].

How does chunking affect RAG answers?

The second limitation is chunking. Cutting out a paragraph separates it from the document and section it came from, and from its title and author. The authors measured this with QASPER-MP: each question about an NLP paper gets the right paper plus ten others, and the question refers to the paper only by author surname ("Smith et al.") or a title fragment ^[1].

Within the same 11-paper pool, giving a long-context model the full papers (LC-EBG) beat the best RAG variant for all five frontier models by 0.05 to 0.13 judge points. Totals looked similar only because RAG gave up on questions where retrieval failed. On answerable questions RAG refused 24–40% of the time, against 13–18% for LC-EBG ^[1].

Figure 2. Re-wrapping retrieved passages in their title and abstract raised scores for four of five models compared with plain concatenation, though still below the full-paper condition ^[1].

The structured variant is the interesting part. The authors compared concatenating the top three passages in rank order (flat) with re-wrapping them in their paper, title and abstract scaffolding and marking the gaps (structured). Opus-4.6 rose from 0.474 to 0.566 and GPT-5 from 0.500 to 0.553, while Sonnet-4.5 alone fell from 0.513 to 0.474. Giving passages back their source identity alone helped models use what was retrieved ^[1].

If an AI answer is right, did it use the evidence correctly?

Answer-only scoring hides right answers resting on the wrong evidence. In a long-context task that needs two pieces of evidence, Gemini-3-Flash (99.5% and 99.8%) and GPT-5 (95.1% and 98.6%) cited both well. GPT-4o cited the first piece 71.3% of the time but the second only 20.9%, and Sonnet-4.5 showed the same lean at 64.6% and 21.4% ^[1].

Figure 3. On the same task, some models cited both pieces of evidence evenly and others cited only one. Answer-only scoring cannot see the difference ^[1].

The gap widened when the evidence in context contradicted what the model already knew. Models reached the expected answer without properly citing the context that should justify it. The authors conclude that correct answers alone can substantially overstate how faithfully a model uses long-context evidence ^[1].

Asking for evidence also helped the answers. Compared with answer-only prompts, prompts requiring citations raised answer accuracy for Sonnet and Gemini models by 9 to 19 points at 300K to 900K characters of context. There was no difference on the easier synthetic task with short, clearly distinct distractors ^[1].

How should a paragraph be written to be cited by AI search?

From here on this is TRAIL's reading. The paper is about building RAG, but if generative search engines pull web pages in as chunks, it carries a lesson for the people writing those pages too.

First, a retrieved chunk is read on its own. If a paragraph says only "this product" or "the company" and the subject sits in the previous paragraph, the lifted chunk no longer says what it is about. That matches the paper's finding that restoring title and abstract raised scores. Second, when an answer needs several pieces of evidence, the more they are scattered across a page, the less likely they are to be retrieved together. Keeping the core evidence for one question in one paragraph works in a page's favour. We explain the gap between being retrieved and being cited in How are retrieval and citation different in AI search?.

What are the limitations of this study?

The authors' own limitations first. The benchmarks are synthetic or semi-synthetic and do not capture the full complexity of real legal, medical or scientific documents, and evidence is scored at the line level rather than the span level. The frontier-model experiments on MultiHop-RAG and QASPER-MP are 50-question pilots. Retrieval variants such as query rewriting, hybrid search and adaptive context selection were not explored and could change these ceilings ^[1].

In the MultiHop-RAG comparison, the long-context condition received all the gold articles while RAG had to search the whole corpus, so the candidate pools differ. The authors state that the 26–34 point gap at top 10 includes the cost of open-corpus retrieval and should not be read as an intrinsic RAG versus long-context difference. The paper does not cover which chunk sizes or retrieval methods real generative search engines use ^[1].

What this means for TRAIL

TRAIL Search's page diagnosis includes a chunk simulator that imitates which part of a page an engine would take. It splits the page into chunks and scores each one for self-containment (the share of pronouns and demonstratives, and whether named entities are present) and answer density (definitions, an early direct answer, numbers and sources). Our writer also checks that each section reads complete on its own, with the direct answer first and its evidence after. The paper's chunking result is the problem those designs target.

The paper also shows a gap. Our simulator scores chunks one at a time, so it does not measure whether the several pieces of evidence one question needs would be retrieved together. For pages aimed at multi-fact questions, a diagnostic that counts how many chunks the required evidence is spread across is the next candidate. Why the same question has to be measured more than once is in How many measurements to trust AI search visibility.

Debopriyo Banerjee et al., "The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering", arXiv 2610.26951 (2026).