Fun-CosyVoice 3.5 - A speech generation model developed by Alibaba Tongyi Labs

What is Fun-CosyVoice 3.5?

Fun-CosyVoice 3.5 is the latest speech generation model released by the Alibaba Tongyi Labs speech team, focusing on multilingual timbre replication and fine-grained expression control. The model's most prominent innovation lies in its FreeStyle natural language control capability—users don't need to master professional parameters; they can directly use conversational commands such as "speak more firmly" or "speak slower" to precisely adjust the speech effect, achieving "free speech generation from a single sentence." Fun-CosyVoice 3.5 adds support for Thai, Indonesian, Portuguese, and Vietnamese, covering 13 languages. The misreading rate of uncommon characters has been reduced from 15.2% to 5.3%, and the first packet latency has been reduced by 35% through Tokenizer frame rate optimization.

Main functions of Fun-CosyVoice 3.5

• FreeStyle Natural Language ControlIt supports direct description of voice effects using conversational commands, such as "make your tone firmer," "lower your tone slightly and speak slower," and "add some emotional fluctuations," allowing for precise adjustment without the need to master professional parameters.
• Multilingual sound reproductionThe app now supports Thai, Indonesian, Portuguese, and Vietnamese, bringing the total to 13 languages. It maintains industry-leading word error rate (WER) and speaker similarity (SpkSim) metrics.
• Improved pronunciation accuracyThe misreading rate of rare characters has dropped significantly from 15.2% to 5.3%, and the reading of long texts is more stable and fluent, with fewer stutters and errors.
• Low latency optimizationThe Tokenizer's frame rate is halved, and the first packet latency is reduced by 35%, making it more suitable for real-time voice interaction scenarios.
• Reinforcement learning-driven optimizationThe language model uses the DiffRO+GRPO strategy to optimize prosody, and the audio generation uses Flow-GRPO technology to improve sound quality and timbre replication similarity.

Technical Principles of Fun-CosyVoice 3.5

• DiffRO + GRPO Prosody OptimizationIn the language model, reinforcement learning is introduced, using a combination of DiffRO (Differential Reward Optimization) and GRPO (Generalized Reward-Penalty Optimization) strategies to add a multi-channel reward mechanism for duration and prosody, thereby improving the naturalness and rhythm of speech.
• Flow-GRPO sound quality enhancementThe Flow-GRPO technique is used in the audio generation part. By combining the Flow Matching framework with reinforcement learning optimization, the similarity of timbre reproduction and the overall audio quality are significantly improved.
• Tokenizer frame rate optimizationHalving the frame rate of the Tokenizer effectively reduces computational overhead, achieving a 35% reduction in first-packet latency while maintaining generation quality.
• End-to-end speech synthesis architectureBased on the CosyVoice series of end-to-end speech synthesis frameworks, it integrates text encoding, acoustic modeling and vocoder to achieve direct generation from text to speech, reducing errors in intermediate steps.
• Multi-task joint trainingBy using a multi-task learning framework, multiple objectives such as speech recognition, timbre cloning, and style control are optimized simultaneously, thereby improving the model's generalization ability in complex scenarios.

How to use Fun-CosyVoice 3.5

• Called through Alibaba Cloud Bailian PlatformAccess the Alibaba Cloud Bailian console, activate the speech synthesis service, obtain the API Key for invocation, and support online debugging and batch generation.
• API Interface CallUsing the official API interface, pass in the text content, the target timbre ID, and the FreeStyle control command (such as "speak softer"), and it will return the generated audio file.

Application Scenarios of Fun-CosyVoice 3.5

• Intelligent customer service and voice assistantLow latency supports real-time conversations, and FreeStyle control can adjust tone of voice according to user emotions, improving service warmth and user satisfaction.
• Audio content creationIn scenarios such as podcasts, audiobooks, and news broadcasts, the reading style can be quickly adjusted through natural language commands, enabling one person to play multiple roles or maintain a unified brand voice.
• Virtual anchors and digital humansThe ability to accurately replicate voice timbre can clone the voice of a specific person, and combined with emotional control, it can generate expressive live/short video dubbing.
• Game and animation voice actingSupports localization in 13 languages, quickly generates character voices, and reduces the cost and time of producing multilingual versions.
• Education and Language LearningIt accurately pronounces uncommon characters to aid teaching, and its multilingual support can be used for foreign language pronunciation demonstrations and oral training.
• Accessibility servicesProvides high-quality voice reading for visually impaired individuals, supporting personalized voice customization and speech rate adjustment.