DragonV2.1 - Microsoft's zero-shot text-to-speech model

What is DragonV2.1?

DragonV2.1 (DragonV2.1Neural) is Microsoft's latest zero-shot text-to-speech (TTS) model. Based on the Transformer architecture, it supports multilingual and zero-shot speech cloning, generating natural and expressive speech from just 5-90 seconds of prompts. The model significantly improves pronunciation accuracy, speech naturalness, and controllability. Compared to DragonV1, its word error rate (WER) is reduced by an average of 12.8%. It supports SSML phoneme tagging and custom dictionaries, allowing for precise control over pronunciation and accent. The model integrates watermarking technology to ensure the compliance and security of speech synthesis.

Main functions of DragonV2.1

• Multilingual supportIt supports over 100 Azure TTS language environments and can synthesize speech in multiple languages to meet the needs of different users.
• Emotional and accent adaptationAdjusting the emotion and accent of the voice according to the context makes the voice more expressive and personalized.
• Zero-sample speech cloningWith just 5-90 seconds of voice prompts, it can quickly generate a user's own AI voice copy, greatly reducing the barrier to voice cloning.
• Quick generationIt can generate high-quality speech synthesis results in a short time with a latency of less than 300 milliseconds and a real-time factor (RTF) of less than 0.05, making it suitable for real-time application scenarios.
• Pronunciation controlIt supports the use of phoneme tags in SSML (Speech Synthesis Markup Language), allowing users to precisely control speech pronunciation through International Phonetic Alphabet (IPA) phoneme tags and custom dictionaries.
• Custom DictionaryUsers can create custom dictionaries and define the pronunciation of specific words to ensure the accuracy of speech synthesis.
• Language and accent controlIt supports the generation of multiple languages and specific accents, such as British English (en-GB) and American English (en-US).
• Watermarking technologyWatermarks are automatically added to the automatically generated speech output, effectively preventing the misuse of speech synthesis content.

Technical Principles of DragonV2.1

• Transformer architectureDragonV2.1 is a deep learning architecture based on the Transformer model, widely used in natural language processing and speech synthesis. The Transformer processes input data based on the self-attention mechanism, which can capture long-range dependencies and generate more natural and coherent speech.
• Multi-head attention mechanismThe multi-head attention mechanism in Transformer allows the model to focus on different parts of the input data from different perspectives, improving the model's ability to capture speech features.
• SSML supportSSML is a markup language used to describe speech synthesis. DragonV2.1 supports phoneme tags and custom dictionaries in SSML. Users can precisely control the pronunciation, intonation, rhythm, etc. of speech through SSML to ensure the accuracy and naturalness of speech synthesis.

DragonV2.1 project address

• Project official websitehttps://techcommunity.microsoft.com/blog/azure-ai-services-blog/personal-voice-upgraded-to-v2-1-in-azure-ai-speech-more-expressive-than-ever-bef/4435233

Application scenarios of DragonV2.1

• Video content creationGenerate multilingual dubbing and real-time subtitles for videos, preserving the original actors' voice styles and enhancing the viewing experience for global audiences.
• Intelligent customer service and chatbotsGenerate natural and expressive voice responses, support multiple languages, improve user experience, and reduce customer service costs.
• Education and TrainingGenerates audio in multiple languages to help language learners practice pronunciation and listening skills, and enhances the interactivity of online courses.
• Smart AssistantIt provides natural voice interaction for smart home devices and in-vehicle systems, supports multiple languages, and improves user convenience.
• Enterprises and BrandsCreate brand voices for advertising and marketing, supporting multiple languages to enhance brand recognition and global market reach.