LongCat-Audio-Codec - Meituan's open-source speech codec solution

What is LongCat-Audio-Codec?

LongCat-Audio-Codec is an open-source speech codec solution from Meituan's LongCat team, specifically designed for large speech language models (Speech LLM). Through a parallel extraction mechanism of semantic and acoustic tokens, it balances semantic understanding with acoustic feature preservation, solving the problem of balancing semantic and acoustic information in traditional solutions. Its low-latency streaming decoder supports real-time interaction, controlling decoding latency to the hundreds of milliseconds level, meeting the needs of scenarios such as in-vehicle voice assistants and real-time translation. It features ultra-low bitrate high fidelity and integrated super-resolution design, achieving high-fidelity audio reconstruction at extremely low bitrates while improving the sampling rate and naturalness of the output audio. LongCat-Audio-Codec provides a one-stop token generator and restorer toolchain, supporting flexible codebook configuration, allowing adjustment of the codebook quantity according to downstream tasks to adapt to different scenarios. Its multi-stage training strategy further optimizes the balance between high compression rate and high audio quality.

Main functions of LongCat-Audio-Codec

• Semantic and acoustic parallel tokenizationIt maps the original audio signal into a sequence of tokens that are both semantic and acoustic, taking into account both the semantic information and acoustic features of speech.
• Low-latency streaming decodingIt adopts a frame-level incremental processing mode to achieve low-latency audio decoding and meet the needs of real-time interaction.
• Ultra-low bit rate high fidelityAchieve high-fidelity audio reconstruction at extremely low bit rates, while integrating super-resolution design to enhance audio naturalness.

The technical principles of LongCat-Audio-Codec

• Semantic-Acoustic Dual-Token Parallel ExtractionSemantic tokens are extracted through a bidirectional Transformer architecture, focusing on the core information of speech; at the same time, acoustic tokens are extracted by combining improved quantization technology to supplement prosody, timbre and other paralinguistic features, thus solving the problem of balancing semantic and acoustic information.
• Low-latency streaming decodingIt adopts a frame-level incremental processing mode to control the dependence on future voice tokens and keep the decoding latency at the level of hundreds of milliseconds, meeting the needs of real-time interaction scenarios.
• Ultra-low bit rate high fidelity and integrated super-resolutionThrough model optimization and training mechanisms, high-fidelity audio reconstruction is achieved at low bit rates; at the same time, the concept of super-resolution is embedded in the decoder to improve the sampling rate and naturalness of the output audio.
• Flexible acoustic codebook configurationIt supports adjusting the number of acoustic codebooks according to downstream tasks to adapt to the needs of different scenarios, such as scenarios with few or many timbres.
• Multi-stage training strategyA multi-stage training strategy was designed to meet the needs of reconstruction under high compression ratio, high sound quality synthesis, and personalized customization.

LongCat-Audio-Codec project address

• Github repository: https://github.com/meituan-longcat/LongCat-Audio-Codec
• Hugging Face Model Libraryhttps://huggingface.co/meituan-longcat/LongCat-Audio-Codec

Application scenarios of LongCat-Audio-Codec

• smart speakerIt enhances the real-time performance and naturalness of voice interaction, enabling speakers to understand and respond to users' voice commands more quickly and accurately.
• In-vehicle voice assistantIt meets the real-time interaction needs in in-vehicle scenarios, provides low-latency voice feedback, and enhances the driving experience.
• Real-time translationWith low-latency streaming decoding support, high-quality real-time speech translation is achieved, reducing latency during the translation process.
• Speech recognition and synthesisIt provides efficient audio processing support for speech recognition and synthesis systems, improving the accuracy of speech recognition and the naturalness of speech synthesis.
• Long audio modelingIt supports efficient encoding and decoding of long audio files, making it suitable for scenarios that require processing long audio content, such as audiobooks and podcasts.
• Multilingual speech processingIt supports multilingual speech processing, providing technical support for cross-language speech applications.