CosyVoice - Alibaba's open-source large-scale speech generation model

What is CosyVoice?

CosyVoice is a large-scale natural speech synthesis model open-sourced by Alibaba's Tongyi Lab. The model supports five languages: Chinese, English, Japanese, Cantonese, and Korean. It can replicate similar timbres, including intonation and emotional details, with just 3 to 10 seconds of audio samples, and can synthesize speech across languages. A key highlight of the model is its ability to finely control the emotion and rhythm of speech through rich text or natural language input, making the synthesized speech lifelike and full of emotion, far surpassing traditional models. It is widely used in voice broadcasting, audiobooks, voice interaction, and other fields, providing users with a highly personalized and emotional voice experience.

CosyVoice 2.0 is an upgraded version of the CosyVoice speech generation model launched by Alibaba Tongyi Labs. The model uses finite scalar quantization technology to improve codebook utilization, simplifies the text-speech language model architecture, and introduces a block-aware causal flow matching model to support diverse synthesis scenarios. CosyVoice 2 has significant improvements in pronunciation accuracy, timbre consistency, prosody, and sound quality, with its MOS score increasing from 5.4 to 5.53. It supports streaming inference and significantly reduces the first packet synthesis latency to 150ms, making it suitable for real-time speech synthesis scenarios.

CosyVoice 3 has been fully upgraded to give voices more emotion and expressiveness. Integrating voice cloning and voice synthesis, it accurately captures a user's timbre, intonation, and speaking habits by requiring only a 3-second voice sample. It supports reading any text aloud using the user's voice. CosyVoice 3 supports nine languages, including Chinese, English, and Japanese, and can generate voices with specific emotions, such as "happy," "sad," and "angry," based on text or commands, making voices more vivid and expressive. CosyVoice 3 preserves the user's accent, making it part of the user's digital identity. The model is now available on Alibaba Cloud's Bailian platform.

Tongyi Labs recently upgraded CosyVoice3 with several improvements, including a 50% reduction in first-packet latency, doubled accuracy in mixed Chinese and English text, support for 9 common languages, 18 dialect accents, and cross-language cloning and emotion control. Furthermore, Fun-CosyVoice3-0.5B was officially open-sourced, providing zero-shot voice cloning capabilities. It allows for voice replication and synthesis of new speech with just 3 seconds of recording time, while also supporting local deployment and secondary development, providing developers with a powerful speech synthesis tool.

Main functions of CosyVoice

• Ultra-low latency streaming speech synthesisIt supports bidirectional streaming speech synthesis, with a first packet synthesis latency of up to 150ms, making it suitable for real-time application scenarios.
• Highly accurate pronunciationCompared to the previous version, the pronunciation error rate has been significantly reduced, especially in handling tongue twisters, polyphonic characters, and rare characters.
• Tonal consistencyMaintaining high consistency of timbre in zero-sample and cross-language speech synthesis, thereby improving the naturalness of the synthesis.
• Natural ExperienceThe rhythm, sound quality, and emotion matching of the synthesized audio have been improved, the MOS score has increased, and it is approaching the level of commercial speech synthesis models.
• Multilingual support: Train on large-scale multilingual datasets to achieve cross-language speech synthesis capabilities.

CosyVoice's technical principles

• LLM backboneBased on a pre-trained large-scale text-based model (such as Qwen2.5-0.5B), the original Text Encoder + random Transformer structure is replaced to perform semantic modeling of text.
• FSQ Speech TokenizerReplace vector quantization (VQ) with full-scale quantization (FSQ) to train a larger codebook (6561), achieving 100% activation and improving pronunciation accuracy.
• Integrated offline and streaming modeling solutionWe propose an integrated modeling scheme that enables both LLM and FM to support streaming inference, achieving rapid synthesis of the first packet audio.
• Upgraded command-controlled audio generation capabilitiesOptimizes the integration of the base model and instruction model, supports emotion, speaking style and fine-grained control instructions, and adds Chinese instruction processing capabilities.
• Multimodal large model technologyBased on multimodal large model technology, it realizes AI technologies such as speech recognition, speech synthesis, and natural language understanding, and provides an intelligent human-computer interaction experience that "can hear, can speak, and understands you".

CosyVoice project address

• Project official website:https://funaudiollm.github.io/cosyvoice2/
• GitHub repository:https://github.com/FunAudioLLM/CosyVoice
• HuggingFace model libraryhttps://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512
• Technical Papers:https://funaudiollm.github.io/pdf/CosyVoice_2.pdf
• CosyVoice 3 Technical Paperhttps://arxiv.org/pdf/2505.17589

Application scenarios of CosyVoice

• Smart assistants and chatbotsProvides natural and fluent voice output for intelligent assistants and chatbots, enhancing the user experience.
• Audiobooks and audiobooksIt generates high-quality audiobooks, supports multiple languages and dialects, and meets the needs of different users.
• Video dubbing and narrationWe provide voice-over services for video content, including educational videos, corporate promotional videos, and dubbing for movies and TV series.
• Customer service and call centerProvide voice interaction in customer service to improve service efficiency and customer satisfaction.
• Language learning and educationIt assists language learning by providing standard pronunciation examples to help learners improve their pronunciation accuracy.