Baichuan-Audio - Baichuan Intelligence's open-source end-to-end voice interaction model
Baichuan-Audio is an end-to-end audio language model developed by Baichuan Intelligence. It supports seamless integration of audio understanding and generation functions, enabling high-quality, controllable real-time Chinese-English bilingual dialogue. Baichuan-Audio is based on multi-code...
What is Baichuan-Audio?
Baichuan-Audio is an end-to-end audio language model developed by Baichuan Intelligence. It supports seamless integration of audio understanding and generation functions, enabling high-quality, controllable real-time Chinese-English bilingual dialogue. Baichuan-Audio uses multi-codebook discretization technology to transform audio signals into discrete symbols, preserving semantic and acoustic information, and enhances audio feature processing capabilities with independent audio headers. The model is based on a two-stage pre-training strategy, combined with interleaved data training, balancing audio modeling and language understanding capabilities. Baichuan-Audio performs exceptionally well in tasks such as real-time speech dialogue, question answering, automatic speech recognition (ASR), and text-to-speech (TTS). The open-source training data and model of Baichuan-Audio provide important resources for speech interaction research.
Main functions of Baichuan-Audio
- Real-time voice dialogueIt supports smooth voice interaction, understands users' voice commands, and generates natural voice responses.
- Speech understanding and generationIt combines automatic speech recognition (ASR) and text-to-speech (TTS) capabilities to achieve seamless conversion from voice input to voice output.
- Multilingual supportIt supports high-quality conversations in both Chinese and English and has cross-language voice translation capabilities.
- Voice Q&AIt can handle complex voice commands and questions, and provide accurate voice responses.
- Audio content generation: Generate aligned speech content based on text guidance to ensure semantic coherence of speech output.
Baichuan-Audio's technical principles
- Audio tokenizationBased on multi-codebook discretization technology, continuous audio signals are converted into discrete audio tokens. Audio features are extracted using Whisper Large Encoder, and semantic and acoustic information are preserved using 8-layer residual vector quantization (RVQ) technology.
- Independent audio headThe model is designed with an independent audio head to process audio tags and enhance the ability to capture audio features.
- end-to-end frameworkThe model uses an end-to-end architecture to process audio input and generate audio output, avoiding the multiple conversions from speech to text and back to speech in traditional cascaded models.
- Two-stage pre-training strategyTo balance audio modeling and language understanding capabilities, Baichuan-Audio employs a two-stage pre-training strategy. The first stage fixes the language model parameters and trains the audio-related components; the second stage releases all parameters for joint training.
- Interleaved data trainingThe model is pre-trained with interleaved data (such as audio-text interleaved data and interleaved text-to-speech data) to enhance cross-modal knowledge transfer and speech generation capabilities.
- Stream Match DecoderA flow-matching-based decoder decodes audio tokens into high-quality Mel spectrograms, which are then used to synthesize natural speech using a HiFi-GAN vocoder.
Baichuan-Audio's project address
- GitHub repository:https://github.com/baichuan-inc/Baichuan-Audio
- HuggingFace model library:https://huggingface.co/baichuan-inc/Baichuan-Audio
- arXiv technical paper:https://arxiv.org/pdf/2502.17239
Application scenarios of Baichuan-Audio
- Real-time voice interactionIt supports fluent voice dialogue, understands voice commands in real time, and generates natural voice responses.
- Voice Q&AIt can handle complex voice commands and questions, and provide accurate voice responses.
- Multilingual supportIt supports high-quality conversations in both Chinese and English and has voice translation capabilities.
- Audio content generation: Generate aligned speech content based on text guidance to ensure semantic coherence.
- Cross-modal capabilityIt combines Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) to achieve seamless conversion from voice input to voice output.