AB
AB
AiBoss

All Categories

project

MuCodec - An ultra-low bitrate music codec jointly developed by Tsinghua University, Tencent AI, and the Chinese University of Hong Kong.

MuCodec is an ultra-low bitrate music codec jointly developed by researchers from Tsinghua University Shenzhen International Graduate School, Tencent AI Lab, and the Chinese University of Hong Kong. It enables efficient music compression and high-fidelity reconstruction. MuCodec...

What is MuCodec?

MuCodec is an ultra-low bitrate music codec jointly developed by researchers from Tsinghua University Shenzhen International Graduate School, Tencent AI Lab, and the Chinese University of Hong Kong. It achieves efficient music compression and high-fidelity reconstruction. MuCodec extracts acoustic and semantic features from music using MuEncoder, discretizes these features using RVQ technology, and reconstructs Mel-VAE features based on stream matching. Finally, it generates reconstructed music using a pre-trained Mel-VAE decoder and HiFi-GAN, achieving industry-leading compression efficiency and sound quality at bitrates from 0.35kbps to 1.35kbps.

MuCodec's main functions

  • Music compressionMuCodec can effectively compress music at extremely low bitrates, supporting a minimum bitrate of 0.35kbps.
  • Music ReconstructionIt can reconstruct high-fidelity music at ultra-low bit rates.
  • Feature extractionMuEncoder is used to extract the acoustic and semantic features of music, capturing its essential characteristics.
  • DiscretizationThe extracted features are discretized based on RVQ (Residual Vector Quantization) technology, which facilitates compression.
  • Stream matching reconstruction: Reconstruct Mel-VAE features using stream matching methods to achieve more refined audio reconstruction.
  • Double bit rate supportIt can operate at both low bit rates (0.35kbps) and high bit rates (1.35kbps), providing flexible application options.

MuCodec's technical principles

  • MuEncoderUsing MuEncoder as a feature extractor, acoustic and semantic features are extracted based on two key aspects of music—vocals and background music.
  • Two-stage training:
    • Phase 1The MuEncoder is constrained to learn and predict masked regions using a masked language model. Based on the unmasked speech signal, it can perceive contextual information and enhance its representation ability.
    • Phase Two: Introducing constraints for reconstruction and lyrics recognition, including the reconstruction of Mel spectrograms and CQT (Constant-Q Transform) features, and lyrics recognition that ensures the extracted features contain semantic information.
  • RVQ (Residual Vector Quantization): RVQ is chosen to discretize MuEncoder features, the representation is compressed based on the residual process, and a more refined approximation is provided by the concatenated codebook.
  • Stream matchingReconstruction is performed using a flow matching-based method, which is more stable to train than GAN (Generative Adversarial Networks) methods and requires fewer training steps to achieve better results in ultra-low bit rate reconstruction tasks. Flow matching uses discretized MuEncoder representations as conditions, and fine-grained reconstruction is performed based on Diffusion Transformer.
  • Mel-VAE decoder and HiFi-GANThe reconstructed Mel spectrogram is recovered through a pre-trained Mel-VAE decoder, and finally the reconstructed music is generated using a pre-trained HiFi-GAN.

MuCodec project address

Application scenarios of MuCodec

  • Online music streaming serviceBy significantly reducing the size of music files while maintaining sound quality, online music streaming service providers can reduce storage and bandwidth costs.
  • Music downloadUsers can download smaller music files, saving storage space and reducing data consumption on mobile devices.
  • Language model constructionWhen constructing language models that require short sequences of music data, effectively compressing the music data improves processing efficiency.
  • Audio editing and processing softwareThe audio editing software integrates MuCodec as an audio compression and reconstruction tool, providing more efficient audio processing capabilities.
  • Mobile devices and embedded systemsIn mobile devices or embedded systems with limited storage and computing resources, reduce resource consumption while maintaining sound quality.
Back