Covo-Audio - Tencent's open-source end-to-end speech model

What is Covo-Audio?

Covo-Audio is an open-source, 7 billion-parameter end-to-end speech model from Tencent, capable of directly processing continuous audio input and generating audio output. Its core innovations include a hierarchical trimodal speech-text interleaved architecture, decoupling technology between intelligence and speaker, and native full-duplex interaction capabilities. Built on Qwen2.5-7B and Whisper, the model achieves state-of-the-art (SOTA) performance in tasks such as spoken dialogue, speech understanding, and audio understanding. As a unified architecture for speech AI, the model avoids the latency and error accumulation of traditional cascaded systems, making it a powerful open-source alternative to GPT-4o speech capabilities.

Main functions of Covo-Audio

• Spoken dialogueSupports natural multi-turn dialogue interaction between end-to-end voice input and voice output.
• Speech understandingThe model deeply integrates acoustic features and semantic content to achieve comprehensive analysis of high-fidelity speech signals.
• Audio understandingThe model supports extension to non-speech scenarios and has a comprehensive perception capability for ambient sounds, music, and other generalized audio.
• Full-duplex interactionIt natively supports low-latency, real-time, two-way voice communication, allowing for natural interruptions and instant responses.

Key information and usage requirements for Covo-Audio

• DevelopersTencent
• Model size7 billion parameters (7B)
• Architecture typeEnd-to-end unified audio language model
• Open source versionCovo-Audio-Chat
• Basic ModelQwen2.5-7B (LLM backbone) + Whisper (audio encoder)
• Model FormatSafetensors, BF16 precision
• paperarXiv:2602.09823
• Open source licenseDedicated License (must be checked in the warehouse)
• Applicable ScenariosResearch and experimental applications
• Python version≥ 3.11 (Recommended)
• Dependency installation:pass requirements.txt One-click installation
• Core dependencies：Transformers, BigVGAN, huggingface-hub
• Hardware resourcesRequires a GPU that supports BF16 inference (sufficient VRAM recommended). Local deployment or cloud inference is both possible.

Covo-Audio's core advantages

• End-to-end unified architectureThe model breaks away from the traditional ASR→LLM→TTS cascade mode, achieving direct audio-to-audio mapping, eliminating error accumulation and significantly reducing inference latency.
• Three-modal deep fusionBy layering and interleaving continuous acoustic features, discrete speech tokens, and natural language text, an effective alignment of high-fidelity prosody and robust semantics is established.
• Decoupling intelligence and timbreThe model leverages multi-speaker training to separate dialogue intelligence from speaker features, supporting flexible transfer and personalized customization of high-quality speech.
• Native full-duplex capabilityThe model uses low-latency streaming processing to achieve real-time two-way interaction, supports natural interruptions and instant responses, and approximates the human conversation experience.
• value of open source ecosystemThe model uses 7 billion parameters to balance performance and cost, and the open complete technology stack lowers the application threshold, providing an independent and controllable foundation solution for Chinese speech AI.

How to use Covo-Audio

• Environmental preparationTo create a Python 3.11 environment and install dependencies, execute `conda create -n covoaudio python=3.11` and `conda activate covoaudio`, and then install the dependencies using `pip install -r requirements.txt`.
• Get codeClone the official GitHub repository to your local machine by running `git clone https://github.com/Tencent/Covo-Audio.git` and then navigate to the project directory using the command `cd Covo-Audio`.
• Download ModelInstall the HuggingFace tool and download the pre-trained weights by executing `pip install huggingface-hub` and `hf download tencent/Covo-Audio-Chat --local-dir ./covoaudio`. The model will be automatically overwritten or saved to the specified directory.
• Configuration pathIf you need to customize the model storage location, modify the `model_dir` and `decode_load_path` parameters in `example.sh` to match the actual path.
• Running inference: Execute the one-click inference script bash example.sh, or modify the audio file path in example.py to implement custom input interaction.
• Custom useReplace the input audio path in example.py with your own file to enable end-to-end voice dialogue interaction with the model.

Covo-Audio project address

• GitHub repositoryhttps://github.com/Tencent/Covo-Audio
• HuggingFace model libraryhttps://huggingface.co/tencent/Covo-Audio-Chat
• arXiv technical paper:https://arxiv.org/pdf/2602.09823

Covo-Audio's Competitive Product Comparison

Application scenarios of Covo-Audio

• Intelligent Customer ServiceThe model supports end-to-end low-latency interaction and full-duplex interruption capabilities, enabling natural and fluent real-time voice Q&A and multi-voice personalized services.
• Smart hardwareThe model can provide offline or cloud-based voice assistant capabilities for smart speakers, in-vehicle systems, and home control systems.
• Content creationSupports efficient generation of multi-character dialogue dubbing, podcast content, and real-time voice translation services.
• Education and Training: Deeply understand the emotional and prosodic details of speech to build an immersive and personalized teaching and interactive system such as oral practice tutoring and virtual lecturer.
• Accessibility servicesIt replaces the visual interface with natural voice interaction, providing visually impaired people and the elderly with a convenient way to obtain information and control devices without typing or touching the screen.