ZipVoice - Xiaomi's zero-shot speech synthesis model

What is ZipVoice?

ZipVoice is a high-efficiency zero-shot speech synthesis (TTS) model released by Xiaomi AI Lab. Based on a Flow Matching architecture, the model includes two versions: ZipVoice (single speaker) and ZipVoice-Dialog (dialogue speech). Through technological innovations such as efficient modeling based on Zipformer, average upsampling strategies, and Flow Distillation methods, the model achieves lightweight modeling and fast inference, addressing the issues of large parameter counts and slow speeds in existing models. ZipVoice-Dialog achieves fast, stable, and natural dialogue speech synthesis through speaker turn embedding vectors and curriculum learning techniques.

ZipVoice's main functions

• Zero-sample speech synthesisIt synthesizes speech with a specific timbre based on input text and reference speech, without requiring a large amount of target speaker data for training.
• Rapid reasoningThrough technological innovations, such as Flow Distillation, the number of inference steps is significantly reduced, the speed of speech synthesis is improved, and it can run efficiently on devices with low computing resources.
• High-quality speech generationWhile maintaining rapid reasoning, the generated speech has high naturalness, good speech quality, and speaker similarity.
• Dialogue speech synthesisThe ZipVoice-Dialog version can synthesize two-person dialogue voices, supports natural and accurate speaker switching, and is suitable for dialogue voice application scenarios such as AI podcasts.
• Open source and scalabilityThe model files, training code, inference code, and OpenDialog speech dialogue dataset are open source, making it convenient for developers to conduct research and expand applications.

ZipVoice's technical principles

• Efficient Modeling Based on ZipformerThis paper introduces the Zipformer architecture into TTS tasks for the first time. It achieves efficient modeling of speech synthesis models by using a multi-scale high-efficiency structure, the collaborative processing of convolution and attention mechanisms, and the multiple reuse of attention weights, which significantly reduces the number of model parameters.
• Average upsampling strategyWe propose an average upsampling strategy. Assuming that each text token has the same duration, we upsample the text tokens and feed them into the speech prediction model. This provides the model with stable initial alignment cues, improving alignment stability and convergence speed.
• Flow Distillation AccelerationBased on the Flow Distillation method, this method combines a pre-trained model with Classifier-free guidance technology to enable the student model to approximate the teacher's prediction through one-step inference without CFG, thereby reducing the number of inference steps and avoiding the additional inference overhead caused by CFG, and achieving fast inference.
• Speaker turn embedding vectorIn dialogue speech synthesis, Speaker-Turn Embedding is introduced to provide the model with fine-grained and accurate speaker identity cues, reducing the difficulty of modeling speaker switching and improving the accuracy of speaker switching.
• Course learning strategiesThe system uses single-speaker speech data for pre-training to enhance speech-text alignment capabilities, fine-tunes on dialogue speech data, learns speaker role switching and natural dialogue speech styles, and solves complex alignment problems in dialogue speech.
• Stereo extensionBy employing techniques such as weight initialization, mono speech regularization, and speaker mutual exclusion loss, ZipVoice-Dialog is extended to enable stereo generation, thereby enhancing the immersive experience of stereo dialogue.

ZipVoice project address

• GitHub repository: https://github.com/k2-fsa/ZipVoice
• HuggingFace model library:https://huggingface.co/k2-fsa/ZipVoice
• arXiv technical paper: https://arxiv.org/pdf/2506.13053

Application scenarios of ZipVoice

• Personal AssistantVoice assistants on devices such as smartphones and smart speakers provide users with a more natural and personalized voice interaction experience.
• In-vehicle voice systemIn cars, it is used for navigation, voice control and other functions to provide a smoother voice interaction.
• audiobooksIt converts text content into speech, generating high-quality audiobooks suitable for novels, news, articles, etc.
• Video dubbingIt automatically generates voiceovers for video content, saving time and costs associated with manual voiceovers and improving content creation efficiency.
• Language learningIt helps learners practice pronunciation using speech synthesis technology and provides standard pronunciation examples.