MHA2MLA - Fudan University, Shanghai AI Lab, and others release methods to optimize LLM inference efficiency.

What is MHA2MLA?

MHA2MLA is a data-efficient fine-tuning method jointly developed by Fudan University, East China Normal University, Shanghai AI Lab, and other institutions. It is based on the introduction of...DeepSeekThe Multi-Head Latent Attention (MLA) mechanism optimizes the inference efficiency of any Transformer-based LLM and reduces inference costs. MHA2MLA is implemented based on two key strategies: first, partial-RoPE, which removes the Rotated Position Encoding (RoPE) dimension of queries and keys that contributes little to the attention score; and second, low-rank approximation, which compresses keys and values based on Joint Singular Value Decomposition (SVD) to reduce the memory footprint of the KV cache. MHA2MLA requires only 0.3% to 0.6% of the original data for fine-tuning, significantly reducing the KV cache (e.g., 92.19%) while keeping performance loss extremely small (e.g., only a 0.5% decrease in LongBench performance).

Main functions of MHA2MLA

• Significantly reduce KV cachingBased on low-rank compression technology, the size of the KV cache is significantly reduced (up to 96.87%), thus reducing memory usage during inference.
• Maintain model performanceFine-tuning with extremely low amounts of data (0.3% to 0.6% of the original training data) keeps performance loss to a very small range (e.g., LongBench performance drops by only 0.5%).
• Compatible with existing technologiesWhen used in conjunction with quantization techniques (such as 4-bit quantization), it can further improve inference efficiency.
• Data efficiencyThe architecture conversion from MHA to MLA can be completed with only a small amount of data, making it suitable for rapid deployment in resource-constrained environments.

The technical principle of MHA2MLA

• Partial-RoPE (Partial Rotational Position Encoding)In MHA, Rotated Position Encoding (RoPE) incorporates positional information into the query vector (Q) and key vector (K) based on rotation operations, helping the model capture positional relationships within a sequence. Based on the contribution of each dimension to the attention score, RoPE dimensions with smaller contributions (i.e., "unimportant" dimensions) are removed, reducing computation and memory usage. This process, called "partial RoPE," preserves key positional information while freeing up space for low-rank compression.
• Low-rank approximationMHA2MLA reduces memory usage by using low-rank joint compression of key-value (KV) matrices. Borrowing from this idea, MHA2MLA performs Singular Value Decomposition (SVD) on the key and value parameter matrices in MHA. This decomposes the key and value matrices into a product of low-rank matrices, approximating the original matrices with fewer parameters. To better preserve the interaction information between keys and values, MHA2MLA employs a joint SVD strategy, performing a joint decomposition of the key and value matrices instead of processing them separately.

MHA2MLA project address

• GitHub repository:https://github.com/JT-Ushio/MHA2MLA
• arXiv technical paper:https://arxiv.org/pdf/2502.14837

Application scenarios of MHA2MLA

• Edge device deploymentReduce model memory usage to make it compatible with resource-constrained smart terminals and IoT devices.
• Large-scale model inferenceReduce key-value cache, improve inference efficiency, and reduce hardware costs and energy consumption.
• Combining quantitative techniquesCombined with quantization technology, it further optimizes inference performance and is suitable for scenarios such as real-time dialogue and online translation.
• Long text processingReduce memory bottlenecks in long text tasks and efficiently process long document summaries and generate long articles.
• Fast model transferWith only a small amount of data for fine-tuning, the MHA model can be quickly converted to an MLA architecture, reducing migration costs.