REFRAG - A high-efficiency decoding framework launched by Meta

What is REFRAG?

REFRAG, developed by Meta Superintendent Labs, is a high-efficiency decoding framework for Retrieval Augmentation (RAG) tasks. It optimizes the way large language models (LLMs) process external knowledge through a "Compress, Sense, Expand" process. REFRAG segments the retrieved long text into multiple "blocks," generating a compact vector representation for each block, thus shortening the input sequence length and reducing computational cost. The model uses a reinforcement learning strategy to intelligently identify key information, preserving the original text of important blocks. The framework significantly improves the latency of first-character generation (up to 30x speedup) while maintaining performance comparable to full-context models, effectively addressing the efficiency problem of large models handling long contexts.

Main functions of REFRAG

• Significantly reduces lead generation delay (TTFT).By optimizing the decoding process, REFRAG can accelerate the first character generation latency by up to 30 times, significantly improving the real-time interactive performance of the system.
• Maintain or improve the quality of generated contentWhile achieving acceleration, REFRAG does not suffer performance loss in perplexity and accuracy across a variety of downstream tasks compared to the baseline model using full context, and even performs better on some tasks.
• Extended Context WindowREFRAG uses compression techniques to enable the model to process more contextual information with the same computational budget. The context window is effectively expanded by 16 times, which helps improve the model's performance in tasks that require long contextual information.
• Adaptable to various application scenariosREFRAG is suitable for RAG tasks and can be applied to other tasks that require processing long contextual information, such as multi-turn dialogues and long document summarization, making it widely applicable.

REFRAG's technical principles

• CompressionThe retrieved long reference materials are divided into multiple "chunks", and a compact vector representation "chunk embedding" is generated for each "chunk". This shortens the length of the input sequence, reduces the amount of subsequent computation, and avoids repetitive encoding calculations.
• Perception (Sense)By training a policy network based on reinforcement learning (RL), we analyze all "block embeddings" and user questions to determine which text blocks contain the most core information and need to be presented to the LLM in their original text form, ensuring that key information is not lost due to compression.
• ExpandThe final input to the main LLM is a mixed sequence containing "block embeddings" of most of the context and a small number of "raw text blocks" that are judged to be key. The LLM generates answers based on optimized input material, retaining key information and minimizing computational load.
• Utilizing the sparsity of attention mechanismsREFRAG observes that the attention mechanism of models in the RAG task exhibits a "block-diagonal" sparse pattern, meaning that the model's attention is mainly focused on the internal structure of a single document and the association between documents and the user's question. REFRAG improves efficiency by selectively compressing and expanding the context, reducing unnecessary computation.

REFRAG's project address

• arXiv technical paper: https://arxiv.org/pdf/2509.01092

Application scenarios of REFRAG

• Retrieval Enhancement Generation (RAG) TaskBy optimizing the decoding process, the delay in generating the first character is significantly reduced, making it suitable for scenarios that require rapid generation of accurate answers, such as intelligent customer service and online question-and-answer systems.
• Multi-turn dialogue systemIn multi-turn dialogues, it efficiently handles long dialogue histories, maintains dialogue coherence and accuracy, and improves user experience.
• Long document summaryREFRAG can effectively process long documents and generate high-quality summaries, making it suitable for automatic summarization of long texts such as news articles and academic papers.
• Knowledge Graph Question AnsweringIt combines knowledge graphs to quickly retrieve relevant knowledge and generate accurate answers, making it suitable for knowledge graph-driven intelligent question-answering systems.
• Content creation assistanceIt enables the rapid generation of creative text in content creation scenarios, helping authors quickly conceive and write articles, stories, etc., thereby improving creative efficiency.