Video-RAG: retrieving OCR and speech text, not more frames

Adding more frames to a long video model can lower its score. Video-RAG extracts on-screen text, speech, and objects, retrieves the parts that match the question, and adds them to the model without any training. With subtitles as the baseline, that lifts the 72B model's Video-MME score from 75.9 to 77.4 ^[1]. The paper appeared at NeurIPS 2025, and the authors are from Xiamen University, the University of Rochester, and other institutions ^[1].

This article reads the arXiv paper (body, Figure 2, and Tables 1 to 5) and checks each number against the tables, as of October 2026 ^[1]. The code is on GitHub, but we did not run or reproduce it. We have no relationship with the authors.

Why doesn't adding more frames work for long videos?

Video models usually sample frames at a fixed interval. More frames should mean more information, but the paper points to two problems ^[1]. First, extra frames repeat similar scenes and add redundancy. Second, past a certain number, more frames can lower performance. LongVA's Video-MME score fell from 52.6% to 51.8% when sampled frames rose from 128 to 384 ^[1][2].

Existing approaches take one of two routes. One retrains the model for long context, which needs large data and GPUs. The other uses a commercial model such as GPT-4o as an agent. The paper says running VideoAgent on the full Video-MME takes about 20 days and about $2,000 of API cost ^[1][3].

What is Video-RAG and how does it work?

Video-RAG adds retrieved text in place of extra frames. The paper describes it as replacing the visual tokens that extra frames would need with auxiliary text pulled from the video ^[1]. It runs in three stages (Figure 2) ^[1].

Stage 1, query decoupling. The model reads only the question, with no frames, and writes a JSON retrieval request: which speech to look for (R_asr), which objects to find (R_det), and how to describe them, such as location, count, and relation (R_type) ^[1].

Stage 2, auxiliary text generation and retrieval. Three databases are built from the video in parallel ^[1].

• OCR: EasyOCR reads on-screen text from each frame. Each chunk is embedded with Contriever^[4] and stored in a FAISS index. Only chunks with similarity above 0.3 are kept.
• ASR: Whisper transcribes the audio and splits it into chunks, indexed the same way as OCR.
• Object detection: keyframes whose CLIP similarity to the object request passes 0.3 are scanned by APE. The objects are converted into scene-graph sentences covering location, count, and relations.

Stage 3, integration and generation. The retrieved text is placed in time order, combined with the question, and fed to the model with the frames. In the paper's notation ^[1]:

Here $F_v$ is the visual feature of the sampled frames, $A_m$ is the set of retrieved auxiliary texts, $Q$ is the question, and $O$ is the answer. The auxiliary text averages about 2.0K tokens, shorter than the roughly 3.0K tokens of full subtitles for the same videos ^[1].

Figure 1. The three stages of Video-RAG: query decoupling, auxiliary text generation and retrieval, then integrated generation ^[1].

What was the experimental method?

On Video-MME, seven open LVLMs were tested: five 7B models and two 72B models. On MLVU and LongVideoBench, only LLaVA-Video 7B and 72B were tested ^[1]. Open-model numbers are the authors' own reproductions. GPT-4o and Gemini numbers are cited from the paper ^[1]. Frame counts follow each model's official setting. Experiments ran on A100 80GB GPUs, and both similarity thresholds were fixed at 0.3 ^[1].

The decoupling prompt was designed for multiple-choice questions, according to the appendix. All three main benchmarks use multiple-choice questions ^[1].

What results did it achieve?

Across the seven Video-MME models, the average gain over the baseline with subtitles was 2.8 points. The 72B LLaVA-Video went from 75.9 to 77.4 with subtitles as the baseline, 0.2 points above GPT-4o with subtitles (77.2) ^[1].

Table 1. Values taken from the paper's Tables 1, 2, and 3 ^[1]. Proprietary numbers are the paper's citations.

Figure 2. Video-RAG against the baseline for the 72B model. Proprietary numbers are the paper's citations ^[1].

The abstract says the 72B model beats Gemini-1.5-Pro. On Video-MME, though, Gemini scores 81.3 with subtitles, above Video-RAG's 77.4. The lead holds on LongVideoBench, 65.4 against 64.0 ^[1]. It does not hold against GPT-4o on LongVideoBench, where GPT-4o scores 66.7 ^[1]. We read each benchmark against its own baseline for that reason.

What do OCR and speech text add to performance?

The component study uses Long-LLaVA 7B with 32 frames on Video-MME ^[1]. With no auxiliary text it scores 52.0. All three text types without retrieval give 59.8, and with retrieval 62.1. Retrieval contributes 2.3 points. The authors report a general improvement, especially on long videos ^[1].

The threshold study shows how strict retrieval changes the result.

Table 2. Values from the paper's Table 5 ^[1]. The 0.3 and 0.0 thresholds differ by 0.1 points, but 0.3 uses 1.9K tokens instead of 3.6K and takes 11 seconds per query instead of 36.

Figure 3. Retrieved text scores higher than the same text added without retrieval, and 0.3 is the best threshold ^[1].

Cost is also reported. For a 7B model it adds about 8GB of GPU memory and about 5 seconds per query. The roughly 2.0K tokens added equal about 14 frames of tokens ^[1].

What are the limits of this paper?

The paper has no dedicated limitations section. The main evaluation covers three multiple-choice benchmarks. The appendix also tests VNBench, a synthetic benchmark. There, objects and OCR text improved retrieval, while the speech text lowered performance from 42.8 to 42.5 ^[1].

We found three limits in reading it. First, the decoupling stage writes object requests from the question alone without seeing frames, so objects the question does not mention are unlikely to be detected. Second, errors from the OCR, speech, and detection tools pass straight into the model, and the paper does not analyze that propagation. Third, the open-model numbers are the authors' reproductions, with no human evaluation. Whether the effect holds for open-ended video QA needs separate testing.

What does this mean for TRAIL?

TRAIL Studio has video analysis. It takes a YouTube or Shorts URL, analyzes it with Gemini, and extracts shot-level cinematography. It also creates SRT subtitles for scenes we produce. It does not have the OCR and speech retrieval structure of this paper.

The design idea still applies to a problem we care about. If what is visible gets turned into text attributes, a query can find it. Whether the same structure helps turn product photos and detail pages into text that AI search can read is a hypothesis. We have not measured it.

When a long video does not fit your model's context, do you spend the budget on more frames or on added text? If you have measured it in practice, we would especially like to hear the results.

Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, Rongrong Ji, "Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension", NeurIPS 2025 (arXiv 2411.13093).

#VideoRAG #RAG #LongVideo #Multimodal #NeurIPS2025 #PaperReview #TRAILLabs