FlexRAG - A high-performance multimodal RAG framework launched by the Chinese Academy of Sciences

What is FlexRAG?

FlexRAG is an innovative retrieval-enhanced generative (RAG) framework that addresses the high computational cost and insufficient generation quality of traditional RAG systems when handling long contexts. It significantly reduces computational burden by compressing retrieved context information into compact embedding representations. FlexRAG's core components include a compression encoder that transforms long contexts into fixed-size embeddings, and a selective compression mechanism that selectively preserves key information by evaluating its importance. This improves the performance of generative models and supports flexible compression ratios and multimodal data processing.

FlexRAG's main functions

• Multimodal RAGFlexRAG supports multimodal RAGs, opening up a wide range of application possibilities for different data modalities.
• Multiple data typesFlexRAG supports a variety of data formats, including text (such as CSV, JSONL), images, documents, web pages, etc., and can flexibly handle various data sources.
• Unified configuration managementBased on Python dataclass With unified configuration management with hydra-core and FlexRAG, configuring RAG processes is much simpler.
• Context compressionFlexRAG transforms retrieved long contextual information into compact embedding representations through a compression encoder, reducing computational burden. This allows the model to process large amounts of data more efficiently.
• Supports multiple searcher typesFlexRAG supports various types of search engines, including sparse search engines, dense search engines, network-based search engines, and multimodal search engines. This allows FlexRAG to be flexibly applied to different data types and scenarios.
• Tips for fine-tuningBy learning a soft prompt, FlexRAG can improve the performance of downstream tasks, enabling the model to better adapt to specific tasks.

The technical principles of FlexRAG

• compression encoder
  • Information extraction and data compressionThe compression encoder is a core component of FlexRAG, responsible for transforming retrieved long contextual information into compact embedding representations. It extracts key information and features to convert contextual information of varying lengths into fixed-size embeddings, thereby reducing the burden on downstream models and improving computational efficiency.
  • Implementation mechanismFlexRAG uses a specific training strategy to enable the compression encoder to learn which information is most important, thereby preserving this key information during compression.
• Selective compression mechanism
  • Importance assessment and dynamic adjustmentThis mechanism prioritizes retaining the most critical information for the generation by assessing the importance of different contextual information. It allows for dynamic adjustment of the retained contextual information based on specific task requirements, making the compression process more flexible.
  • Compression ratio allocationTo balance compression performance and preservation of contextual information, FlexRAG groups the contexts based on their estimated importance and assigns a different compression ratio to each group.
• Two-stage training workflow
  • Pre-training and fine-tuningFlexRAG training consists of two phases: pre-training and fine-tuning. The pre-training phase is performed on large-scale datasets to establish the model's basic language understanding and generation capabilities; the fine-tuning phase is performed on task-specific datasets to optimize the model's performance on those specific tasks.

FlexRAG project address

• Github repository:https://github.com/ictnlp/flexrag

Application scenarios of FlexRAG

• Open Domain QuestionsWhen faced with complex problems in unknown areas, FlexRAG can generate accurate and detailed answers by retrieving information from relevant knowledge bases.
• Dialogue systemIn multi-turn conversations, FlexRAG can retrieve relevant information based on historical conversation content and generate coherent and insightful responses.
• Document summarization and generationBased on information from the knowledge base, FlexRAG can better extract and synthesize key information from documents, generating high-quality summaries.
• Knowledge-intensive tasksIn tasks requiring extensive background knowledge, such as natural language reasoning and text classification, FlexRAG can improve the accuracy and reliability of the model by retrieving external knowledge.
• Multimodal content processingFlexRAG supports the integration of various data types, including text, images, and documents, and can be applied to the generation and processing of multimodal content.