RAG answers improve more from verification than retrieval

RAG answers improve more from verification than retrieval

A full factorial test of 16 RAG feature combinations: one answer check beat three retrieval features

Across 16 combinations of four RAG features, a post-generation completeness check alone beat three retrieval features combined, at double the latency. RAG features are usually validated one at a time against a baseline but deployed together. This paper ran all 16 on/off combinations of four features to measure their interactions.

By · Fred Kim
RAGAnswer verificationAI searchEvaluation

RAG answer quality rose more from features that verify the generated answer than from features that refine retrieval. In an experiment that ran all 16 on/off combinations of four RAG features and scored 768 answers, a configuration with only a completeness check, which finds missing facets after the answer is written and retrieves again, scored 4.31 out of 5. The configuration with three retrieval-stage features combined scored 4.11. In exchange, mean latency more than doubled, from 78 seconds at baseline to 169 seconds (paper, Table 3)[1].

The source is a single-author arXiv paper posted in 2026 by Ng S. T. Chong of the Campus Computing Centre at United Nations University in Tokyo[1]. The raw data for 760 scored answers and the definitions of all 24 queries are published on Zenodo[2]. We did not reproduce the experiment, and the figures come from the arXiv v2 text as of November 2026. The queries were written by one person and the scoring was done by a single LLM judge, so the numbers hold for this corpus and these implementations.

We build a product that measures how often brands appear in AI search answers, and its writing feature also runs checks and a revision after generation. That gives us a stake in how much a verification step is worth. The study we covered yesterday dealt with the upstream decision of whether to search at all; this paper deals with the downstream step of checking the answer after retrieval and generation. It reads well alongside our post on why RAG fails to retrieve all the evidence it needs.

RAG features are validated one by one and switched on together

RAG pipeline enhancements are usually shown to work one at a time against a baseline. In production several features run at once, yet nobody measures whether they overlap or cancel each other out. To fill that gap, the author used a 2⁴ full factorial design covering every on/off combination of four features (paper, Section 1)[1]. Running all 16 combinations gives each feature's main effect independent of the others and exposes interactions when two or more run together.

The main effect of feature is the mean quality of the eight configurations with it on minus the mean of the eight with it off, and effect size is Cohen's d, that difference divided by the pooled standard deviation.

Here is composite quality, the unweighted mean of accuracy, completeness and usefulness (each scored 1 to 5), and is the pooled standard deviation of the on and off groups. Confidence intervals come from 5,000 bootstrap resamples.

How to evaluate combinations of RAG pipeline features

Four features were tested. Two add evidence at the retrieval stage without LLM calls, one splits the query before retrieval, and one verifies the answer after generation (paper, Table 1)[1].

FeatureStageWhat it doesExtra LLM calls
Section expansion (SE)RetrievalPulls sibling chunks under the same heading pathAlmost none
Agentic search (AS)Pre-retrievalAn LLM splits a complex query into sub-queries and retrieves for each1 plus a retrieval per sub-query
Completeness check (CC)Post-generationSplits the question into facets, checks each against the answer, retrieves again for missing facets and rewrites1 to 2 per hop
ToC-guided retrieval (ToC)RetrievalMatches the query against pre-embedded table-of-contents entries and loads whole page sectionsAlmost none

Table 1. The four RAG features tested. Source: Chong 2026, Table 1[1].

The corpus is five public documents: GDPR (78 pages), the UN Staff Rules (118 pages), the EU AI Act (144 pages), the UNOPS Procurement Manual (195 pages) and NIST SP 800-53 Revision 5 (492 pages). These are regulatory and policy texts with heading hierarchies of varying depth and dense cross-references[1].

The 24 queries fall into eight types: simple queries the baseline handles, four types targeting one feature each, two types targeting a pair of features, and a type requiring all four. Where "What is the definition of personal data under GDPR?" is a simple query, "What are the conditions for lawful processing under GDPR and what makes consent valid?" is a completeness query demanding an exhaustive list (paper, Appendix D)[1]. The design crossed 16 configurations, 24 queries and two models (DeepSeek-V4-Flash and grok-4.3) into 768 conditions, and 760 completed after 8 transient API failures.

Claude Opus 4.6 scored every answer at temperature 0, called directly rather than through the pipeline. Using reference answers built from the source documents and manually checked for facts as the rubric, it scored accuracy, completeness and usefulness from 1 to 5 each (paper, Section 3.3 and Appendix C)[1]. Retrieval used nomic-embed-text embeddings, 512-token chunks, hybrid search blending cosine similarity and BM25 at 0.7 to 0.3, and reranked 20 candidates down to the top 5.

Verification had a larger effect than retrieval optimization

The completeness check (CC) had the largest effect of the four. It raised composite quality by 0.39 points (d = +0.48, p < 0.001) and was significant on accuracy, completeness and usefulness alike. ToC-guided retrieval raised quality by 0.18 points with no added LLM calls (d = +0.22, p = 0.003), agentic search (AS) was borderline (d = +0.14, p = 0.050) with no accuracy effect, and section expansion (SE) was zero on every score (paper, Tables 4 and 7)[1].

FeatureOnOffDifferencedp
Completeness check (CC)4.243.85+0.39+0.48<0.001
ToC-guided retrieval (ToC)4.133.95+0.18+0.220.003
Agentic search (AS)4.103.98+0.12+0.140.050
Section expansion (SE)4.044.04-0.01-0.010.894

Table 2. Main effect of each feature on composite quality (1 to 5, 759 answers). Source: Chong 2026, Table 4[1].

The configuration view makes it plainer. CC alone (4.31) beat every configuration without CC, including SE, AS and ToC all switched on (4.11). The two models averaged 4.05 and 4.03, statistically indistinguishable, so the result does not lean on one model (paper, Section 4.1)[1]. Mean quality rose with the number of features enabled, but the spread within each count was wide, so which features were on mattered more than how many.

How to verify RAG answers: split the question into facets

CC splits the question into information facets and judges each as covered, missing or partial in the answer. If any facet is missing or several are partial, the answer is marked incomplete, and only the missing facets are retrieved again before the answer is rewritten. It also uses the heading path to check for gaps in numbered provisions (paper, Section 3.1)[1]. Loops were capped at two hops and three sub-queries. Revisiting a generated answer has precedents such as Self-RAG[3], and CC is a variant that breaks that judgment down by facet.

The author stresses the judging prompt. A holistic prompt asking "is this answer complete?" leaned lenient and produced many false negatives, and asking facet by facet cut them substantially. The facet judge was a local llama3.1:8b (paper, Section 2 and Appendix A)[1]. The structure worked even with a judge far smaller than the cloud models writing the answers.

CC raised accuracy as well as completeness (+0.25 points). The author reads this as the re-retrieval cycle correcting errors while filling gaps. The effect was largest on queries demanding exhaustive lists, reaching d = +0.83 on completeness (paper, Section 4.4)[1]. Paired with ToC, CC's rewrites per query fell from about 1.6 to between 1.3 and 1.5, a trace of pages fetched up front by ToC covering facets CC would otherwise flag. Our post on RAG hallucination defense covered the limits of automated verification; this experiment is a case where splitting verification into a facet checklist eased part of that limit.

Testing features one at a time flips their ranking

One-at-a-time comparisons against the baseline would have reported SE +0.02, AS +0.18, CC +0.71 and ToC +0.29. The factorial design also exposed a redundancy: adding SE to any configuration with CC erased even that small gain (paper, Section 5)[1]. AS and ToC act at different stages (pre-retrieval and retrieval) and their effects simply added up, while every three-way and four-way interaction was indistinguishable from zero. Feature effects added rather than multiplied.

Query type changed the ranking too. CC's overall d = +0.48 averages a range from +0.16 on simple queries to +0.83 on completeness queries. AS raised quality by 0.63 points on queries needing both structural retrieval and completeness, yet lowered it by 0.22 points on the very queries designed to target AS (paper, Section 4.4)[1]. The author suggests that splitting the query can fragment retrieval instead of improving it. Evaluating on simple queries alone undervalues CC and overvalues AS. It is the same lesson as why AI search visibility should not be measured once: one measurement design choice can change the conclusion.

How to improve RAG performance: configurations by latency budget

Plotting quality against latency, 6 of the 16 configurations are Pareto-optimal. All four features on scored highest at 4.35 but took 175 seconds, more than double the 78-second baseline, with CC alone right behind at 4.31 and 169 seconds. Configurations with CC averaged 210 seconds against 92 without it, making CC the main source of latency (paper, Section 4.5 and Table 8)[1].

ConfigurationComposite qualityLatency (seconds)
Baseline3.6078
SE+ToC3.9080
AS+ToC4.0995
SE+AS+ToC4.11103
CC4.31169
SE+AS+CC+ToC4.35175

Table 3. The six Pareto-optimal configurations on quality and latency (averaged over both models and 24 queries). Source: Chong 2026, Table 8[1].

The author's advice depends on the latency budget. Turn CC on by default where quality matters, start with ToC where latency is tight because it adds no LLM calls, and use AS+ToC (4.09 at 95 seconds) as the balanced option. SE could be dropped on this corpus (paper, Section 5 and Table 6)[1]. To cut CC latency, the author proposes as future work a query-complexity gate that skips CC on queries the baseline already answers well. In our view, the hidden-state classifier from the study showing agents know in advance whether they need to search is a candidate for that gate.

What this means for AI search answers

From here on this is our interpretation. AI search such as ChatGPT search or Perplexity is also a RAG setup that writes answers from retrieved documents, and a step that checks for missing facets and retrieves again after answering would open the citation pool a second time. The paper does not examine whether commercial AI search uses such a step, so what follows is inference resting on the assumption that the structure is the same.

Under that assumption, the implication for pages is coverage at the facet level. Because CC retrieves again only for missing facets, a page with a passage that answers one facet of a question precisely (the conditions for valid consent, say, or how fines are calculated) is more likely to be picked up on re-retrieval. Where the study measuring how much cited pages are absorbed into answers looked at what happens after citation, this experiment shows that a step checking whether the answer used its evidence fully separates good answers from weak ones. Splitting a question into facets also resembles how AI fans a question out into several searches.

What this study did not cover

The author lists the limits of scale and design. One person wrote all 24 queries and marked the relevant pages alone, and each query type has only two to four unique queries. Repeating the same queries over 16 configurations and two models raised the answer count, but the bootstrap intervals do not correct for clustering at the query level (paper, Limitations)[1]. Scoring relies on a single LLM judge with no human agreement check or second judge. Because answers are scored against references, valid perspectives absent from the reference do not count.

Generalization is narrow as well. The two answering models are cloud models of similar capability, so effects may differ on small local models, and the corpus is only public regulatory and policy text, so technical manuals or financial reports may behave differently. All four features are single implementations written by the author, so the results describe these implementations more than the techniques in general. Page-level retrieval accuracy was measurable only for ToC, and ToC pages hit relevant content on 21% of runs (paper, Section 4.6)[1]. One question of our own: even the baseline took 78 seconds per answer, so absolute latency is hard to compare with real-time search services.

In practice, turn on verification first and mix query types in evaluation

For teams building RAG, the order is simple. If latency allows, add a facet-level completeness check after generation first; if latency is tight, start with structure-based retrieval (table of contents and heading hierarchy), which adds no LLM calls. When choosing features, run combinations instead of toggling one at a time, and mix simple queries with exhaustive-list queries in the evaluation set so the ranking does not flip.

Our writing feature follows the same shape. After drafting, it runs rule, chunk and vocabulary checks and makes one local revision targeting only the failed checks. Reading this paper gave us something to weigh: whether that check list should include "is there a passage answering each facet of the question?"

Frequently asked questions

To improve RAG performance, should I work on retrieval or answer verification first?

In this experiment, verification came first. A configuration with only a completeness check, which tests after generation whether each facet of the question made it into the answer, scored 4.31 out of 5, above the configuration with three retrieval features combined (4.11). Mean latency rose from 78 to 169 seconds, so a latency-bound service is better off starting with table-of-contents retrieval, which adds no LLM calls.

How do you implement a completeness check for RAG answers?

Split the question into information facets, judge each facet as covered, missing or partial in the answer, then retrieve again only for missing facets and rewrite the answer. A single prompt asking whether the answer is complete judged leniently and missed gaps, while asking facet by facet cut those misses substantially. In the paper, an 8B local model handled the facet judgments.

Can't I just test RAG features one at a time and pick the winners?

You miss the overlaps. Here, section expansion did almost nothing on its own, and the little it did vanished once the completeness check was on. Agentic search raised quality on some query types and lowered it on others, so an evaluation set built from one query type flips the ranking.

Does this apply directly to AI search like ChatGPT?

Not directly. The experiment used five public regulatory and policy documents, 24 queries, two cloud models of similar capability, and the author's own feature implementations. Whether commercial AI search verifies answers and retrieves again after generating is outside the paper's scope.

References

  1. [1]Ng S. T. Chong, "Post-Generation Verification Dominates Retrieval Optimization: A 2^4 Factorial Ablation of RAG Pipeline Features", arXiv:2609.35774v2, 2026
  2. [2]Raw data (760 scored responses) and the 24 query definitions, Zenodo
  3. [3]Akari Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection", ICLR 2024

Summary

  • RAG features are usually validated one at a time against a baseline but deployed together. This paper ran all 16 on/off combinations of four features to measure their interactions.
  • A post-generation completeness check that tests whether each facet of the question made it into the answer had the largest effect (d = +0.48), and enabling it alone beat enabling three retrieval features together.
  • Table-of-contents retrieval improved quality without extra LLM calls, agentic search changed sign depending on query type, and section expansion had no effect.
  • The highest-quality configurations more than doubled baseline latency, so the right configuration depends on the latency budget.
  • With 24 queries written by one person and a single LLM judge, the results hold for this corpus and these implementations.

Check this topic against your own brand

TRAIL Search measures how ChatGPT and Perplexity answer your customers' questions and finds and fixes where the brand is missing, turning this post into a diagnosis. Start with 10 questions, no card required.

More posts

Keep reading on this topic

This post is part of the Papers category, which collects all 44 posts on the topic. See all posts in Papers