RedKnot - Xiaohongshu's open-source long-text reasoning acceleration engine

What is RedKnot?

RedKnot is Xiaohongshu's open-source long-text LLM inference acceleration engine. It decouples the KV Cache by the attention head dimension, classifying it offline into global heads that require global recalculation and local heads that can be reused locally. Combined with SegPagedAttention, it achieves paginated storage by head and fusion of variable-length attention kernels, avoiding kernel penalties caused by attn_mask. RedKnot also introduces sparse FFN, performing dense computation on tokens with high attention scores to solve the bottleneck of short-context FFN.

RedKnot's main functions

• Head classification sparseOffline, each pair is categorized into global headers that require global recalculation and local headers that can be locally reused, ensuring stability between requests without online overhead.
• Location-independent KV multiplexingIt supports reusing the pre-computed KV cache when the same document fragment appears in a non-prefix position, breaking the traditional prefix matching limitation.
• Sparse FFN accelerationDense FFN computation is performed on the top-k tokens with the highest attention scores, while the remaining tokens follow the residual identity path, directly reducing the bottleneck of short context FFN.
• SegPagedAttention storageChange the KV Cache from dense tensor to segmented pagination storage, physically retaining only the tokens actually needed for each header.
• Elastic sparsity recoveryThe shallow layer uses conservative local attention plus dense FFN to protect the early residual flow, while the deep layer uses global head recalculation and sparse FFN to balance accuracy and efficiency.
• Architecture-independent runtimeIt supports different attention architectures such as GQA, MoE, and MLA through four adapter interfaces: PROFILE, BUILDSTATE, SELECTVISIBLESTATE, and EXECUTE.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

RedKnot's technical principles

• Decoupling KV Cache by HeaderTraditional systems treat KV cache as a dense [B,H,L,D] tensor. RedKnot found that the effective context range and importance of different attention heads are highly differentiated, so he split the cache system along the head dimension.
• Head-level sparsity replaces token-level sparsityToken-level recovery requires taking the union of important tokens from each header, which leads to bloat. RedKnot changed to recovery per header, recalculating only a few global headers and directly reusing local headers, thus avoiding the precision-latency dilemma.
• FFN and Attention Orthogonal OptimizationIn short contexts (2–8K), FFN accounts for 57–62% of TTFT, which cannot be reached by attention optimization; RedKnot selects important tokens to execute FFN through attention signals, forming a multiplicative superposition benefit with KV sparsity.
• Paged storage eliminates mask penaltyDense layouts with attn_mask disable the FlashAttention fast path, resulting in a 4.9–7.6x kernel penalty. SegPagedAttention uses head-based pagination and is combined with the varlen kernel, without constructing a mask throughout the process.
• Tiered elasticity strategyThe model has a high proportion of shallow local heads and weak semantic selectivity, so conservative recovery prevents error propagation; the proportion of deep global heads increases and attention is more focused, so the sparse strategy yields the greatest benefit and the least loss of accuracy.

How to use RedKnot

• Offline portrait classificationRun on the target model PROFILE The interface, tested offline via needle-in-a-haystack, will test each (layer, head) For headers classified as global or local, a stable Head Class Map is generated for reuse in subsequent requests with zero overhead.
• Pre-built reusable state:use BUILDSTATE The interface pre-computes frequently used document fragments as a KV cache and stores them in the Global/Local KV Pool in pages according to the header dimension, realizing location-independent offline pre-construction.
• Dynamic state selectionWhen an online request is received, via SELECTVISIBLESTATE Based on the query semantics and the Head Class Map, the interface selects the local header key-value pairs that need to be reused and the global header range that needs to be recalculated from the cache pool.
• Fusion reasoning execution: call EXECUTE The interface performs a full attention recalculation on the global header and writes it to the Online KV Cache. The local header directly reuses the paginated KV and performs local attention. At the same time, for low-scoring tokens, the FFN calculation is skipped and the residual path is used.
• Service-oriented deployment and integrationFrom GitHub repository https://github.com/rednote-machine-learning/RedKnot Pull the source code based on SGLang and connect it to the existing inference service stack according to the four major adapter interface specifications to go online.

RedKnot's core advantages

• Granular alignment of head-to-head decouplingBreaking through the traditional token-level dense KV cache abstraction, it unifies the granularity of storage, computation, and recovery to the attention head dimension, matching the actual head-sparse structure of the workload.
• Offline portrait with zero online overheadThe classification of global and local headers is highly stable across requests, requiring only one offline profiling operation. When running online, the application directly looks up the table without adding any inference latency.
• Head-level recovery replaces token-level recoveryOnly about 12–15% of the global header is recalculated, and 85–88% of the local header is directly reused, avoiding the runaway recalculation caused by token-level union expansion, while eliminating cascading error propagation.
• Sparse FFN orthogonal acceleration: Perform dense FFN on the top-k tokens with high attention scores, and take the residual path for the rest. This directly reduces the FFN bottleneck that accounts for 57-62% of TTFT in short contexts, and forms a multiplicative superposition benefit with attention optimization.

RedKnot's project address

• GitHub repositoryhttps://github.com/rednote-machine-learning/RedKnot
• arXiv technical paper: https://arxiv.org/pdf/2606.06256

Comparison of RedKnot's similar products

Application scenarios of RedKnot

• RAG Long Document Q&ABy splicing tens of thousands of search fragments into the prompt and using position-independent key-value reuse and head-level sparse recovery, the pre-filling delay for long texts is reduced from tens of seconds to several seconds.
• Programming Agent Multi-round Tool InvocationBy continuously calling the tool dozens of times and accumulating historical context, the bottleneck of FFN, which accounts for more than half of the TTFT, is directly reduced by using sparse FFN.
• Long session memory systemBy unifying user memory, tool output, and historical state into a long context, and reusing them head-to-head, the concurrency of a single card can be increased from 4 to more than 30.
• Multi-Agent Collaboration FrameworkMultiple agents can dynamically exchange and rearrange context fragments, and position-independent key-value reuse breaks the "must prefix match" restriction, avoiding repeated pre-filling.
• Real-time streaming long text generationLocal headers directly reuse recent pagination key-value pairs, while global headers are recalculated as needed, maintaining low first-word latency and high generation stability even in a 128K context.