AB
AB
AiBoss

All Categories

project

SVDQuant - A diffusion model post-training quantization technique from MIT.

SVDQuant is a post-training quantization technique developed by an MIT research team. For diffusion models, it quantizes the model's weights and activation values to 4 bits, reducing memory usage and accelerating the inference process. SVDQuant introduces high-precision low-rank branch absorption...

What is SVDQuant?

SVDQuant is a post-training quantization technique developed by an MIT research team. For diffusion models, it quantizes the model's weights and activation values to 4 bits, reducing memory usage and accelerating the inference process. SVDQuant introduces a high-precision low-rank branch to absorb outliers during quantization, achieving 3.5x memory optimization and 8.7x latency reduction on a 16GB 4090 GPU while maintaining image quality. SVDQuant supports the DiT architecture, is compatible with the UNet architecture, and can seamlessly integrate with off-the-shelf low-rank adapters (LoRAs) without requantization, providing an effective solution for deploying large diffusion models on resource-constrained devices.

SVDQuant's main functions

  • Quantization compressionThe weights and activation values of the diffusion model are quantized to 4 bits, reducing the model size and memory usage.
  • Accelerate reasoningQuantization reduces computational complexity and improves the inference speed of models on GPUs.
  • Low-rank branches absorb outliersIntroducing low-rank branches to handle outliers in quantization reduces quantization error.
  • kernel fusionThe design of the Nunchaku inference engine is based on kernel fusion to reduce memory access and further improve inference efficiency.
  • Supports multiple architecturesA diffusion model compatible with DiT and UNet architectures.
  • LoRA integrationSeamless integration of low-rank adapters (LoRAs) without the need for requantization.

SVDQuant's technical principles

  • QuantificationPerforming 4-bit quantization on the model's weights and activation values poses a challenge to maintaining model performance.
  • Outlier handlingThe smoothing technique is used to transfer outliers in the activation values to the weights. The weights are decomposed into low-rank components and residuals based on SVD.
  • low-rank branches: Introducing a 16-bit precision low-rank branch to handle outliers in the weights, quantizing the residuals to 4 bits, and reducing the difficulty of quantization.
  • Eckart-Young-Mirsky theoremRemove dominant singular values from the weights, significantly reducing the magnitude and outliers of the weights.
  • Nunchaku Inference EngineThe inference engine is designed based on a kernel that integrates low-rank branches and low-bit branches, reducing memory accesses and kernel call counts, and lowering latency.

SVDQuant's project address

Application scenarios of SVDQuant

  • Mobile devices and edge computingDeploying diffusion models on smartphones, tablets, or other mobile devices reduces model size and increases inference speed, enabling faster image generation and processing.
  • Personal computers and workstationsImprove the efficiency of image and video generation on personal computers or workstations, suitable for content creators and professional designers.
  • Cloud computing platformDeploy SVDQuant in cloud services to provide online users with fast image generation services, such as virtual try-on, image editing, and enhancement.
  • Interactive applicationsIn applications that require real-time feedback, such as games, virtual reality (VR), and augmented reality (AR), reducing latency provides a smoother user experience.
  • Low power devicesEnables more energy-efficient model inference in Internet of Things (IoT) devices and other low-power devices.
Back