AngelSlim - Tencent Hunyuan's open-source multimodal large model compression toolkit

What is AngelSlim?

AngelSlim is a self-developed and open-source full-modal large model compression toolkit from Tencent's Hunyuan team. Through techniques such as quantization, speculative sampling, sparsification, and distillation, it provides efficient and easy-to-use compression and acceleration solutions for large language models (LLM), visual language models (VLM), and speech models. The tool integrates mainstream compression strategies such as FP8/INT8/INT4 quantization, GPTQ, and AWQ, supports one-click invocation and end-to-end deployment, and the trained models can be seamlessly integrated with inference frameworks such as vLLM and Sglang. AngelSlim has significantly upgraded its speculative sampling training framework, pioneering the Eagle3 architecture, extending speculative sampling technology to full-modal scenarios for the first time. This allows smaller models to learn to draft multi-step candidate tokens for larger models, which are then validated in parallel. Real-world testing shows inference speed improvements of up to 1.4-1.9 times. Currently, it supports compression optimization for mainstream models such as Hunyuan, DeepSeek, and the Qwen series. Developers can use it to... pip install angelslim Quick installation and use.

AngelSlim's main functions

• Multi-precision quantization compressionIt supports multiple precision quantization methods such as FP8, INT8, and INT4, and integrates mainstream quantization algorithms such as GPTQ and AWQ, significantly reducing model storage and computational overhead.
• Speculative sampling accelerationIt pioneered the Eagle3 training architecture, enabling small models to draft multi-step candidate tokens for large models, which are then validated in parallel by the large models, resulting in an inference speed improvement of up to 1.4-1.9 times.
• Full modal coverage supportIt supports compression and acceleration of large language models (LLM), visual language models (VLM), and speech models (ASR/TTS), and for the first time realizes the application of speculative sampling in all modal scenarios.
• thinning and distillationIt integrates structured/unstructured sparsity, knowledge distillation and other techniques to further compress the model size while retaining performance.
• One-click compression callIt provides a highly integrated API interface, simplifying the calling process of mainstream compression algorithms and lowering the threshold for developers to use it.
• End-to-end deployment and integrationThe trained model can be directly used in mainstream inference frameworks such as vLLM and Sglang, achieving a seamless transition from compression to deployment.
• Multi-model ecosystem compatibilityIt already supports compression optimization for mainstream open-source models such as Hunyuan, DeepSeek, Qwen, Qwen2.5VL, and Qwen3-Omni.

AngelSlim's technical principles

• Quantization compression principleBy mapping model weights from high-precision floating-point numbers (FP16/FP32) to low-precision integers (INT8/INT4) or FP8 format, storage and computation are reduced. At the same time, algorithms such as GPTQ (Ground-by-Ground Quantization Optimization) and AWQ (Activation-Aware Weight Quantization) are used to minimize accuracy loss.
• Speculative Decoding PrincipleThe system trains a small draft model to generate multiple candidate tokens in advance, and then a large target model verifies them in parallel. It accepts valid tokens, rejects incorrect tokens, and rolls back for correction. The collaborative mechanism of "small model drafting + large model overseeing" breaks through the bottleneck of serial decoding.
• Eagle3 Architecture PrinciplesBased on standard speculative sampling, a "forward-looking" training strategy is introduced, which allows the draft model to learn and predict the distribution characteristics of tokens in future multiple steps, improves the quality of candidate sequences, and thus increases the acceptance length of large models (up to 1.8-3.5 times), achieving a higher speedup ratio.
• Knowledge distillation principleThe output distribution and intermediate layer features of the large model (teacher model) are transferred to the small model (student model). Through soft label supervision and feature alignment, the small model can inherit the capabilities of the large model while compressing its size.
• sparsity principleBy pruning redundant weight connections (unstructured sparsity) or entire neurons/attention heads (structured sparsity), the computational density of the model is reduced, and inference speed is increased in conjunction with hardware acceleration.
• Full-modal unified framework principleTo address the differences in model architecture across different modalities (text, image, and speech), a unified compression interface and modality adaptation layer are designed to enable cross-modal reuse of techniques such as quantization and speculative sampling.

AngelSlim's project address

• GitHub repositoryhttps://github.com/Tencent/AngelSlim
• Hugging Face Model Libraryhttps://huggingface.co/collections/AngelSlim

Application scenarios of AngelSlim

• Accelerating Large-Scale Model Inference in the CloudIt provides quantization compression and speculative sampling acceleration for large models with hundreds of billions of elements, such as Hunyuan, DeepSeek, and Qwen, reducing GPU memory usage and computation latency, and improving service throughput in high-concurrency scenarios.
• Edge model deployment optimizationIt compresses large models to a size suitable for running on mobile phones, tablets, and IoT devices through INT4/INT8 quantization, supporting offline inference and privacy protection scenarios.
• Accelerating Multimodal AI ApplicationsIt provides a full-modal compression solution for image and text understanding of visual language models (VLM) and real-time transcription and synthesis of speech models (ASR/TTS), accelerating applications such as intelligent customer service, content moderation, and real-time translation.
• AIGC content generation efficiency improvementIn generative scenarios such as AI writing, code generation, and image description, speculative sampling technology can significantly reduce the latency of the first token and the overall generation time, thereby improving the user experience.
• Enterprise privatization deployment reduces costsIt helps enterprises compress large open-source models and deploy them to private clouds or local servers, reducing computing power procurement costs and achieving cost-effective AI platform construction.