IndexCache - A sparse attention acceleration technology jointly developed by Tsinghua University and Zhipu. - AiBoss

What is IndexCache?

IndexCache is a sparse attention acceleration technology developed by Tsinghua University and Zhipu team. Addressing the high computational overhead of indexers in DeepSeek Sparse Attention (DSA), it reduces redundant computation by reusing indexes across layers. IndexCache discovered that the overlap rate of top-k tokens selected by adjacent layers is as high as 70%-100%, therefore dividing layers into "full layers" (computing and caching the index) and "shared layers" (directly reusing the cache). This method can eliminate 75% of indexer computation, achieving a 1.82x speedup in pre-filling and a 1.48x speedup in decoding within a 200K context scenario, with almost no loss in model performance. Its effectiveness has been verified on a 30B parameter model and a 744B parameter GLM-5 model.

Main functions of IndexCache

Cross-level index reuse: Use the top-k indexes of adjacent layers with a high overlap rate of 70%-100% to allow shared layers to directly reuse the cached indexes of the full layer, avoiding redundant calculations.
Significantly reduce indexer overheadIt can remove 75% of the indexer computation, retaining only 1/4 of the indexers to maintain model performance.
Significantly accelerates reasoningAchieve 1.82x speedup for pre-filling and 1.48x speedup for decoding in a 200K context, reducing user wait time.
Zero additional memory overheadReuse is achieved through a conditional branch, without the need to allocate additional GPU memory.
Two deployment options are provided.The no-training scheme determines the optimal layer pattern through greedy search, while the training-aware scheme optimizes the indexer parameters through multi-layer distillation loss.
Production-level verificationIt has been validated on 30-parameter models and 744-parameter GLM-5, and supports SGLang and vLLM inference frameworks.

The technical principles of IndexCache

Cross-level index similarity discoveryThe research team discovered through heatmap analysis that the top-k token sets output by the indexers of adjacent layers in the DSA model have extremely high similarity, with an overlap rate generally between 70% and 100%, indicating that a large amount of indexing computation is redundant.
Layered Role Division MechanismIndexCache divides the model layer into two categories: the Full Layer retains the original indexer and is responsible for calculating and caching the latest top-k index; the Shared Layer no longer runs its own indexer and directly reuses the index cached by the most recent Full Layer for sparse attention calculation.
Dynamic mode selection strategyFor trained models, a greedy search algorithm based on calibration data is used to try to convert each layer into a shared layer and evaluate the impact on the model output, while retaining key layers as full layers. For training from scratch scenarios, a multi-layer distillation loss is introduced, allowing each full layer indexer to learn to serve the needs of multiple subsequent shared layers simultaneously.
Reasoning process optimizationDuring the inference process, each layer only adds a simple condition judgment, and switches between calculating a new index and reusing the cached index according to a preset mode, realizing cross-layer sharing of the indexer without modifying the model architecture or adding extra storage.

Key information and usage requirements for IndexCache

Proposing organizationDeveloped jointly by Tsinghua University and Z.ai.
In response to the problemIt addresses the computational bottleneck of the indexer in DeepSeek sparse attention scenarios with long contexts, accounting for up to 81% of the pre-filling time when dealing with 200K tokens.
Core PrinciplesBased on the high overlap rate of 70%-100% of the top-k indexes of adjacent layers, redundant calculations are reduced by reusing them across layers.
acceleration effectRetaining 1/4 of the indexer achieves a 1.82x speedup in pre-filling and a 1.48x speedup in decoding.
performance loss: Almost no loss of quality, and even better performance on some reasoning tasks.
Validation ModelThe results were validated on both the 30B parameter DSA model and the 744B parameter GLM-5 model.
Hardware RequirementsRequires an NVIDIA GPU (such as H100), but no additional video memory is needed; it reuses standard DSA memory space.
Software environmentSupports SGLang or vLLM frameworks and provides ready-made patches that can be directly used for DeepSeek-V3.2, GLM-5 and other models.
No training programThis method is applicable to pre-trained DSA models and requires a small batch of calibration data to run a greedy search to determine the optimal layer mode.

The core advantages of IndexCache

Significant accelerationSupports 1.82x faster pre-filling and 1.48x faster decoding in 200K context, significantly reducing user wait time.
Zero performance lossAfter removing 75% of the indexer computation, the model quality remains almost unaffected, and even shows a slight improvement in some tasks.
Zero additional overheadA conditional branch can be reused without increasing GPU memory usage, reusing the memory already allocated by the standard DSA.
Plug and playProvides SGLang and vLLM patches, which can be directly applied to mainstream models such as DeepSeek-V3.2 and GLM-5 without modifying the model architecture.
Flexible deploymentIt supports both no-training and training-aware approaches, adapts to both trained models and de novo training scenarios, and allows for flexible configuration of the indexer retention ratio.
Production-level verificationIt has been verified to be effective on the GLM-5 large model with 744B parameters and has the capability for large-scale deployment.

The project address for IndexCache

GitHub repositoryhttps://github.com/THUDM/IndexCache
arXiv technical paper: https://arxiv.org/pdf/2603.12201

Comparison of IndexCache and similar competing products

Comparison Dimensions	IndexCache	Native DSA	Full Attention Anchor Method
Core Mechanism	Top-k indexes output by cross-layer indexer reuse	Each layer runs a lightweight indexer independently.	Reuse index based on full attention anchor layer
Computational overhead	Removing 75% of indexers speeds up pre-filling by 1.82x.	In a 200K context, the indexer accounts for 81% of the pre-fill time.	The full attention layer needs to be retained, which incurs high computational costs.
Applicable Scenarios	DSA architecture that completely eliminates full attention	Standard DSA Deployment	Architecture requiring full attention as an anchor point
Implementation complexity	One if/else branch, zero extra video memory	Standard implementation	An anchor layer strategy needs to be designed.
Training requirements	Supports training-free deployment or training-aware optimization	Complete training required	Joint training is usually required.
Production Validation	744B GLM-5 Verification	DeepSeek-V3 Production Applications	Mostly small to medium-sized experiments

Application scenarios of IndexCache

Long document processingSuitable for scenarios such as academic paper reading and legal contract analysis, it improves pre-filling speed by 1.82 times in 200K contexts, significantly reducing the time users wait for the first token.
Multi-step reasoning taskIt supports complex logical chain reasoning such as mathematical proofs and code generation, and the decoding speed is increased by 1.48 times, accelerating the process of generating thought chains.
Agent WorkflowIt enables multi-turn tool calls, autonomous task planning, and other agentic processes, reduces the cost of long context reasoning, and supports more complex agent interactions.
RAG SystemUsed for enhancing the generation of large-scale knowledge base retrieval, efficiently handling the long context integration and generation of web-scale retrieval results.
Real-time dialogue serviceIt is suitable for online services such as customer service robots and intelligent assistants, which can increase throughput, reduce serving costs, and improve the end-user experience.