Delta-CoMe - A novel incremental compression algorithm open-sourced by Tsinghua University in collaboration with OpenBMB and other universities.

What is Delta-CoMe?

Delta-CoMe is a novel incremental compression algorithm proposed by the Tsinghua University NLP Lab in collaboration with the OpenBMB open-source community, Peking University, and Shanghai University of Finance and Economics. An 80GB A100 GPU can easily load up to 50 7B models, saving approximately 8 times the GPU memory, while maintaining model performance almost identical to the fine-tuned model before compression. Delta-CoMe combines low-rank decomposition and low-bit quantization techniques, utilizing the low-rank characteristics of the model parameter increment (Delta) to achieve mixed-precision compression. This method significantly reduces storage and inference costs while maintaining almost no loss in model performance, performing exceptionally well in handling complex tasks such as mathematical, coding, and multimodal tasks.

Main functions of Delta-CoMe

• Model compressionBased on mixed precision compression technology, it significantly reduces the storage and memory requirements of large language models (LLMs), making it possible to deploy more models on limited hardware resources.
• Performance retentionThe goal is to maintain model performance while compressing, especially in complex tasks such as mathematical problem solving, code generation, and multimodal tasks, ensuring that the performance of the compressed model is close to that of the uncompressed fine-tuned model.
• MultitaskingIt supports the simultaneous deployment of multiple models with different capabilities, making it suitable for multi-tenant and multi-task processing scenarios, thus improving the flexibility and efficiency of model deployment.
• Improved reasoning speedBased on the implementation of the Triton kernel operator, the inference speed is improved by nearly 3 times compared to the PyTorch implementation, further optimizing the model's running efficiency.

The technical principle of Delta-CoMe

• Low-rank decompositionUsing singular value decomposition (SVD) to perform low-rank decomposition of the model parameter increment (Delta), we found that the Delta parameter has low-rank characteristics, that is, most of the changes are concentrated on a few principal components.
• Hybrid precision quantizationBased on the results of low-rank decomposition, a mixed-precision quantization method is adopted to assign different bit widths to different singular vectors. Singular vectors corresponding to larger singular values are represented with higher precision, while singular vectors corresponding to smaller singular values are represented with lower precision, thus reducing storage requirements.
• Long-tail distribution utilizationDelta-CoMe observes that the singular values of the Delta parameter exhibit a long-tailed distribution, meaning that most singular values are small, with only a few being large. Therefore, more aggressive compression is applied to singular vectors corresponding to small singular values, while higher precision is preserved for singular vectors corresponding to large singular values.
• Compatibility and generalizabilityThe Delta-CoMe method is suitable for specific models or tasks and has good generalization ability. It is compatible with different backbone models (such as Llama-2, Llama-3 and Mistral) and maintains good performance on a variety of tasks.
• Hardware optimizationTo further improve inference speed, Delta-CoMe implements the Triton kernel operator for mixed-precision quantization, making hardware deployment more efficient.

Delta-CoMe project address

• GitHub repository:https://github.com/thunlp/Delta-CoMe
• arXiv technical paper:https://arxiv.org/pdf/2406.08903

Application scenarios of Delta-CoMe

• Multi-tenant serviceIn a cloud computing environment, it supports providing a dedicated model for each customer on limited hardware resources, saving video memory and computing resources.
• MultitaskingIn scenarios where multiple different tasks need to be processed simultaneously, models for different tasks can be effectively compressed and deployed.
• End-side deploymentReduce the storage and memory footprint of models on resource-constrained edge computing devices.
• Model fine-tuning serviceFor models that require frequent fine-tuning to adapt to new data or tasks, compress the fine-tuned model increments to reduce storage and inference costs.
• Academic research and developmentResearchers and developers can compress and deploy large language models for experimentation and development, without being limited by hardware resources.