ViDoRAG - A visual document retrieval enhancement and generation framework jointly developed by Tongyi, USTC, and Shanghai Jiao Tong University.

What is ViDoRAG?

ViDoRAG is a visual document retrieval enhancement and generation framework developed by Alibaba Tongyi Lab in collaboration with the University of Science and Technology of China and Shanghai Jiao Tong University. Based on multi-agent collaboration and dynamic iterative reasoning, it addresses the limitations of traditional methods in retrieval and reasoning when handling complex visual documents. ViDoRAG employs a multimodal retrieval strategy using a Gaussian Mixture Model (GMM) to dynamically adjust the number of retrieval results and optimize the integration of textual and visual information. The framework includes three agents: Seeker, Inspector, and Answer, responsible for rapid filtering, detailed review, and final answer generation, respectively. It iterative interaction progressively refines the answer, improving generation quality and consistency. ViDoRAG significantly outperforms existing methods on the ViDoSeek benchmark dataset, achieving an average performance improvement of over 10%, demonstrating its efficiency and superiority in visual document retrieval and reasoning tasks.

ViDoRAG's main functions

• Multimodal retrievalIntegrating visual and textual information to achieve accurate document retrieval.
• Dynamic Iterative ReasoningMulti-agent collaboration (Seeker, Inspector, Answer Agent) gradually refines the answer, improving the depth and accuracy of reasoning.
• Complex document comprehensionSupports single-hop and multi-hop inference, and can handle complex visual document content.
• Generate consistency guaranteeThe Answer Agent ensures the accuracy and consistency of the final answer.
• High-efficiency generationDynamically adjust the number of search results to reduce computational overhead and improve generation efficiency.

ViDoRAG's technical principles

• Multimodal hybrid retrievalThis approach combines textual and visual retrieval results and dynamically adjusts the number of retrieval results based on a Gaussian Mixture Model (GMM). The GMM fits the similarity distribution between the query and the document set, dynamically determining the optimal number of retrieval results (Top-K). This avoids the noise and computational overhead associated with fixed-number retrieval, effectively integrates visual and textual information, improves retrieval accuracy, and reduces interference from irrelevant information.
• Dynamic Iterative Reasoning Framework:
  • Seeker Agent: Responsible for quickly filtering relevant images or document fragments to provide global clues.
  • Inspector Agent: Conduct a detailed review of the screening results and provide feedback or preliminary answers.
  • Answer AgentIntegrate the initial answer from the Inspector, verify consistency, and generate the final answer.
• Coarse-to-fine generation strategyStarting from a global perspective, the focus gradually shifts to local details. Multi-agent collaboration enables a coarse-to-fine generation process, reducing interference from irrelevant information and improving generation efficiency and accuracy.
• Reasoning ability activatedBased on iterative reasoning and multi-agent collaboration, the model's reasoning ability is activated, especially when dealing with complex visual documents, improving the model's performance in multi-hop reasoning and complex document understanding tasks.
• Dynamic retrieval length adjustmentBased on GMM, the number of search results is dynamically adjusted to avoid the limitations of a fixed Top-K value, reduce computational overhead, and improve search efficiency and generation quality.

ViDoRAG's project address

• GitHub repository:https://github.com/Alibaba-NLP/ViDoRAG
• arXiv technical paper:https://arxiv.org/pdf/2502.18017

ViDoRAG Application Scenarios

• EducationIt helps students and teachers quickly retrieve charts, data, and text content from textbooks, generating accurate answers and summaries of key knowledge points.
• Financial industryExtract key data and charts from financial reports and market research documents to generate analytical reports and assist in investment decisions.
• HealthcareQuickly locate charts and data in medical literature to assist doctors in conducting research or generating patient education materials.
• legal industry: Retrieve relevant clauses and case charts from legal documents to assist lawyers in analyzing cases or preparing documents.
• Enterprise knowledge managementExtract key information from internal documents, quickly answer employee inquiries, and generate project reports or risk analyses.