project
VimRAG - An open-source, multimodal knowledge base and RAG framework from Alibaba Tongyi.
VimRAG is an open-source, full-modal RAG framework from Alibaba's Tongyi Labs, supporting mixed knowledge bases of text, graphs, and videos. The framework innovatively uses a multimodal memory graph (DAG) instead of linear context, modeling reasoning as a dynamic directed acyclic graph to achieve pathfinding...
What is VimRAG?
VimRAG is an open-source, multimodal RAG framework from Alibaba's Tongyi Labs, supporting hybrid knowledge bases of text, graphs, and videos. The framework innovatively uses a multimodal memory graph (DAG) instead of linear context, modeling reasoning as a dynamic directed acyclic graph to achieve path backtracking. VimRAG optimizes through graph guidance strategies, precisely pruning invalid paths and intelligently allocating visual tokens to solve cross-modal association breaks and state blind spots.
VimRAG's main functions
-
Full-modal knowledge base retrievalIt unifies the processing of mixed knowledge bases of text, images, and videos, supports cross-modal content association and retrieval, and eliminates the need to convert video OCR into subtitles or create separate databases.
-
Dynamic memory graph (DAG)The system replaces the linear context with a directed acyclic graph, and each node encapsulates "text summary + visual evidence + topological location" to enable backtracking and trial-and-error inference paths.
-
Graph-guided strategy optimization (GGPO)Based on graph topology, fine-grained contribution evaluation is performed, invalid nodes (dead ends) are automatically pruned, high-value retrieval paths are retained, and training gradient variance is reduced.
-
Intelligent visual energy distributionVisual tokens are dynamically allocated based on node importance. Core evidence retains high-resolution images, while edge nodes are downgraded to text descriptions or directly pruned to save computing power.
-
Retrieval-Perception DecouplingIt separates "retrieval actions" from "visual perception," supports progressive information acquisition from coarse-grained to fine-grained, and avoids cross-modal association breakage.
-
Multi-round iterative reasoningThe agent can autonomously decide the next search target (deepen the video search or go back to search the text), and avoids deadlocks of repeated queries through branching and trial and error.
VimRAG's technical principles
- Multimodal memory graph (DAG)The system upgrades the traditional linear context to a dynamic directed acyclic graph. Each node encapsulates "text summary + visual evidence + topological location," with the root node representing the user query. Inference paths are generated through iterative expansion. The system supports branching and trial-and-error, automatically marking redundant paths as dead ends, retaining critical links, and completely resolving the "state blind spot" (forgetting already queried content as the context expands).
- Retrieval-Perception DecouplingThe process separates the "thinking retrieval" and "visual perception" stages. The agent first determines the retrieval action (search, summarize, answer), and then performs fine-grained perception (region selection, cropping, scaling) on the returned multimodal content, achieving a progressive information acquisition from coarse-grained to fine-grained.
- Graph-guided strategy optimization (GGPO)Fine-grained contribution evaluation is performed based on memory graph topology. During training, it can accurately backtrack: pruning non-contributing dead-end nodes in positive samples (masking gradients), and protecting nodes that were retrieved validly but incorrectly in negative samples (avoiding penalties). This significantly reduces gradient variance and accelerates policy convergence.
- Dynamic allocation of visual energyThe "energy value" is calculated based on the importance of a node in the graph (topological out-degree, time decay, priority score). High-energy nodes retain the complete visual token, while low-energy nodes are downgraded to sparse representations or plain text descriptions, enabling complete cross-modal understanding with extremely low computational cost.
How to use VimRAG
- API Quick Start GuideBy calling the Qwen3.5-Plus model through the Alibaba Cloud DashScope interface and configuring the API Key, the Streamlit interactive interface can be launched with one click, allowing for question and answering within a preset knowledge base that combines text, images, and videos.
- Local deployment (requires A100 80G VRAM)Deploy the Qwen2.5-VL-7B model locally and start the service via vLLM, while also starting the search engine API. This is suitable for scenarios that require private deployment or custom models.
- Build your own knowledge baseOrganize images, PDFs (converted to images), and videos (segmented into segments) into a corpus; construct a vector index using a GVE or Qwen3-VL Embedding model; launch the search service API, connect the custom knowledge base to the VimRAG Agent, and begin searching for questions and answers.
Key information and usage requirements for VimRAG
- Product PositioningThe full-modal RAG framework, open-sourced by Alibaba Tongyi Labs, is designed for enterprise-level mixed knowledge bases of text, images, and videos and has been integrated into Alibaba Cloud Bailian Knowledge Base.
- Core InnovationThe system replaces the linear context with a multimodal memory graph (DAG) and achieves fine-grained contribution evaluation through graph-guided strategy optimization (GGPO). Combined with an intelligent visual energy allocation mechanism, it solves the problems of cross-modal association breakage and "state blind zone".
- Performance indicatorsIn the unified mixed corpus test, Qwen3-VL-8B achieved an average accuracy of 50.1%, which is significantly better than Vanilla RAG (37.6%) and ReAct (37.7%).
- Hardware environmentAPI mode does not require a local GPU; local deployment requires an NVIDIA A100 with 80GB of video memory.
- Software DependencyPython 3.10 requires the installation of the dependencies listed in requirements.txt.
- Access credentialsTo use the API mode, you need to obtain the Alibaba Cloud DashScope API Key in advance.
VimRAG's core advantages
- Unified processing of all modesIt natively supports a mixed knowledge base of text, images, and videos, eliminating the need to convert videos into subtitles or create separate libraries, thus fundamentally solving the problem of broken cross-modal associations.
- Structured memory graph (DAG): Replace linear context stacking with dynamic directed acyclic graphs, with each node encapsulating text summary, visual evidence, and topological location, enabling backtracking and trial-and-error inference paths.
- Graph-guided strategy optimization (GGPO)Based on graph topology, fine-grained contribution evaluation is performed, ineffective dead-end paths are automatically pruned and high-value nodes are protected, significantly reducing training gradient variance and accelerating convergence.
- Intelligent visual energy distributionVisual tokens are dynamically allocated based on the importance of nodes in the reasoning topology. Core evidence is preserved as high-definition images while peripheral nodes are downgraded to text, using extremely low token consumption to carry the complete understanding process.
- Retrieval-Perception Decoupling DesignThe "retrieval action" and "visual perception" modules are separated, supporting progressive information acquisition from coarse to fine granular, completely eliminating the "state blind spot" and dead loop of repeated queries in traditional solutions.
VimRAG's project address
- GitHub repositoryhttps://github.com/Alibaba-NLP/VRAG
- HuggingFace model libraryhttps://huggingface.co/papers/2602.12735
- arXiv technical paper: https://arxiv.org/pdf/2602.12735v1
Key information and usage requirements for VimRAG
| Comparison Dimensions | VimRAG | ReAct | MemAgent/Mem1 |
|---|---|---|---|
| Architecture Design | Structured topology of dynamic directed acyclic graph (DAG) | "Think-Action-Observation" linear pipeline | The memory mechanism is shallow in structure and relies on implicit learning. |
| Context Management | Nodes encapsulate text summaries, visual evidence, and topological locations, supporting path backtracking. | Each step simply splices new content into the context, without any structural connection. | Linear or shallow memory management, lacking explicit topological relationships |
| Cross-modal processing | Explicitly modeling multimodal associations and achieving cross-modal verification through graph structures. | Easily forgets checked modalities and associations, resulting in "state blind spots". | Cross-modal association relies on implicit learning in the model, resulting in weak association. |
| Training optimization | Graph-guided strategy optimization (GGPO), fine-grained contribution assessment, and precise pruning of dead ends. | No specific optimization mechanism, relies on end-to-end learning | A one-size-fits-all approach to rewards and punishments based on the final answer results in a large gradient variance. |
| Problem solved | Supports branching and trial and error, automatically identifies and removes invalid paths, and avoids duplicate queries. | It is easy to get stuck in an infinite loop of repeatedly generating similar queries. | Valid nodes that are difficult to distinguish between exploratory search and conclusive verification |
VimRAG Application Scenarios
- Smart manufacturingIt integrates technical documents, design drawings, and training videos to enable cross-modal search. When engineers inquire about design changes, it can automatically link meeting minutes, drawing annotations, and video discussion clips.
- Online EducationThe system integrates course recordings, textbooks, and blackboard notes, allowing students to simultaneously view video feeds, formula screenshots, and text explanations when asking questions about concept derivations.
- Enterprise KnowledgeThis approach integrates meeting minutes, PowerPoint presentations, and training videos to address the cross-modal disconnect where text mentions a diagram but the diagram itself cannot be found.
- e-commerce retailIt integrates product details, real-life photos, and introductory videos, and simultaneously extracts video footage and instruction manual images and text when users inquire about installation steps.
- Media contentFor long video footage libraries, reporters can accurately locate relevant footage and narration timestamps when searching for events.