MMDocRAG: can AI pick evidence in multimodal document RAG?

Given long documents mixing figures and tables, AI had to pick the evidence for its answer. Even assuming perfect retrieval, the best model reached only 70.2% quote-selection F1. Of 20 candidates, only a median of 2 are the real evidence, and text evidence (52.0) was harder than image evidence (80.7) ^[1]. MMDocRAG appeared at NeurIPS 2025 Datasets & Benchmarks, and the authors are from Huawei Noah's Ark Lab ^[1].

This article reads the arXiv paper and the NeurIPS camera-ready, Tables 1, 2, 3, 5, 6, and Figure 8, as of October 2026, checking each number against the tables ^[1]. The project page has the data and code ^[2], but we did not run or reproduce it. We have no relationship with the authors.

What does a multimodal document RAG benchmark measure?

Enterprise documents are not body text alone. They mix financial tables, charts, and diagrams, and one question's answer is scattered across pages and formats. MMDocRAG makes AI do two things on such documents. First, pick the evidence quotes for the question (quote selection); second, weave the picked quotes into an answer that mixes text and images (multimodal generation) ^[1].

The benchmark has 4,055 expert-annotated QA pairs, split into 2,055 for development and 2,000 for evaluation. The questions draw on 222 long documents spanning 10 types, averaging 67 pages and about 33,000 words. Answers may interleave text and images ^[1].

The scoring hinges on how candidate quotes are mixed. Each question gets 15 or 20 candidate quotes, of which only a median of 2 (mean 2.7) are gold; the rest are hard negatives. Hard negatives are decoy quotes that look similar to the question or answer in text or image, drawn from top retrieval results. That lets the benchmark measure, by F1, whether a model can isolate the real evidence among many candidates ^[1].

Figure 1. The task and scale of MMDocRAG. Selecting evidence from 20 candidates, even perfect retrieval left the best model at 70.2% F1 ^[1].

Can AI pick evidence from figures and tables in documents?

Start with quote selection, and it is hard even under perfect retrieval. To remove retrieval error, the candidate set is fixed to include the gold quotes and only selection is measured; in this setting, the best model, GPT-4.1, scored 70.2% F1. The authors note that even state-of-the-art models reach "only 70.2%" ^[1].

The gap by type is large. GPT-4.1 scored 52.0 on text-quote selection and 80.7 on image quotes. It picks image evidence fairly well but struggles more with text evidence. Answer quality is scored on five criteria (fluency, citation quality, text-image coherence, reasoning, and factual accuracy), where GPT-4.1 led at 4.14 out of 5 ^[1].

The split between proprietary and open models was clear too. The paper reports that proprietary VLMs significantly outperform open VLMs and LLMs. For open and small models, multimodal input became a liability: a same-size LLM beat its matching VLM on selection F1 by 175.9% at 7B and 62.7% at 32B in the Qwen family. Smaller models struggle more with long multimodal inputs ^[1].

In document RAG, is retrieval or quote selection the bottleneck?

So far we assumed perfect retrieval. In practice a retriever must fetch the candidates first, and that stage was a large bottleneck. In the end-to-end run with GPT-4.1, the oracle selection F1 of 70.2 fell to 54.4 with a default retriever (BGE). Even expanding the query into multiple clauses and combining four retrievers (BGE, Qwen3-0.6B, BM25, E5) to raise recall to 84.9% reached only 59.9 ^[1].

Table 1. Values from the paper's Table 6 (GPT-4.1, 20 candidates). Even recovering 85% of recall, selection and answer quality stayed below perfect retrieval ^[1].

Difficulty also split by format. Text-quote retrieval recall was 34.6-49.5% across settings, while image quotes ran 77.8-86.8%, far higher. Text evidence was the harder side in both retrieval and selection ^[1].

Figure 2. Swapping perfect retrieval for real retrievers lowers the score. Text-quote recall (34-49%) below image (78-87%) made retrieval the bottleneck ^[1].

Should images be fed as OCR text or VLM-described text?

There are two ways to pass a document image to a model: extract its characters with OCR, or convert the figure into text a VLM describes. Comparing the two, VLM-described text was better. Quote-selection F1 rose from 59.8 with OCR text to 66.3 with VLM-described text on average ^[1].

The reason was length and information. VLM descriptions were 0.5x longer than OCR for tables and 2.8x longer for figures. OCR tends to miss layout and relations inside a figure, while VLM descriptions captured more of that multimodal information ^[1].

Position mattered for selection too. Although MMDocRAG spreads gold quotes evenly across candidate positions, models still favored earlier evidence, echoing the trend reported in Lost in the Middle. A gold quote in the first position was most likely selected, and the chance fell toward the end, especially for image quotes ^[1][3].

Figure 3. Feeding the same image as VLM-described text rather than OCR characters raised selection F1 (59.8 vs 66.3 on average) ^[1].

How was the experiment designed?

The evaluation is broad. It covers 60 off-the-shelf models (23 proprietary and 37 open) plus 9 fine-tuned on the MMDocRAG dev set. It compares 14 retrievers: 6 text, 4 visual, and 4 hybrid ^[1].

Scoring has two axes. Quote selection matches picked quotes against gold quotes by F1, and answer quality is scored on five criteria by an LLM judge. Candidates were tested at 15 and 20 quotes; 20 candidates averaged about 3.6K tokens. Hard negatives were decoys with high similarity to the question or answer, drawn from top retrieval results ^[1].

What are the limits of this benchmark?

Three points worth noting. First, scoring answer quality with an LLM judge carries the limits of automated metrics; the authors aimed beyond them, but judge-model bias remains. Second, the 222 documents span 10 types, but whether they skew toward certain languages or domains needs a closer look at the distribution in the paper. Third, drawing hard negatives from top retrieval results reflects current retrievers, so difficulty can shift as retrievers change. The benchmark is best read as measuring difficulty at a point in time ^[1].

What this means for TRAIL

The paper's conclusion touches the problem we work on in AEO/GEO. "Selection is hard even with perfect retrieval" means getting the evidence in front of the model is not the end; how that evidence is expressed so the AI actually picks it matters.

The result that stands out is that an image fed as VLM-described text rather than OCR characters is selected more. TRAIL Search diagnoses whether brand pages get cited in AI search, which runs in the same direction as the hypothesis that information held only in figures and tables is hard for AI to read. Whether turning the visual information in product photos and detail pages into text attributes AI can read raises citation likelihood is ours to measure. We have not confirmed it in numbers yet.

When you put documents mixing formats into AI search, which gets cited more: unpacking the figure and table information into text, or leaving the images as they are? If you have compared it in practice, we would like to hear the result.

Kuicai Dong, Yujing Chang, Shijie Huang, Yasheng Wang, Ruiming Tang, Yong Liu, "Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering" (MMDocRAG), NeurIPS 2025 Datasets & Benchmarks (arXiv 2505.16470).

#MMDocRAG #MultimodalRAG #DocumentQA #DocVQA #NeurIPS2025 #PaperReview #TRAILLabs