Muyan-TTS - An open-source text-to-speech model for zero-shot speech synthesis.

What is Muyan-TTS?

Muyan-TTS is an open-source text-to-speech (TTS) model designed for podcast scenarios. Pre-trained on over 100,000 hours of podcast audio data, the model achieves zero-shot speech synthesis, generating high-quality speech without requiring extensive target speaker data. The model supports speaker adaptation for personalized speech customization. Muyan-TTS boasts fast synthesis speed, generating 1 second of audio in 0.33 seconds, making it suitable for real-time applications. Muyan-TTS can naturally and coherently synthesize long content such as podcasts and audiobooks, and supports local deployment and API usage, facilitating integration into various applications.

Main functions of Muyan-TTS

• Zero-sample speech synthesisIt generates high-quality speech with a small amount of reference speech and text, without requiring a large amount of target speaker data.
• Speaker adaptationPersonalized voice customization is achieved by fine-tuning based on a small amount of target speaker voice data.
• Quick generationIt generates 1 second of audio in 0.33 seconds, making it suitable for real-time and batch generation of long audio content.
• Long content coherent synthesisIt supports the natural and coherent synthesis of long content, such as podcasts and audiobooks.
• Offline deployment friendlySupports local inference, ensuring data privacy and low latency.

The technical principle of Muyan-TTS

• Framework DesignBased on the GPT-SoVITS framework, a pre-trained Llama-3.2-3B language model (LLM) is used, combined with the SoVITS model for audio decoding. The LLM is responsible for aligning text and audio tokens to generate intermediate representations, and the SoVITS model decodes the intermediate representations into audio waveforms.
• Data processingThe dataset contains over 100,000 hours of podcast audio data, processed through multiple stages including data collection, cleaning, and formatting to ensure high quality and diversity. Audio was transcribed into text using an Automatic Speech Recognition (ASR) model, and audio embeddings were quantized into discrete tokens to form a parallel corpus.
• Pre-training and fine-tuningLLM is pre-trained on parallel corpora to learn the relationship between text and audio tokens. Supervised fine-tuning (SFT) is then used to further optimize the model with a small amount of speech data from the target speaker, improving the naturalness and similarity of the synthesized speech.
• Decoder optimizationBased on the VITS fundamental model as the decoder, the illusion problem is reduced, and the stability and naturalness of speech generation are improved. The decoder is fine-tuned on high-quality audio data to further enhance the fidelity and expressiveness of the synthesized speech.
• Inference accelerationHighly efficient memory management and parallel inference technology improve inference speed and reduce latency. Supports API mode with automatic acceleration, making it suitable for real-time applications.

Muyan-TTS project address

• GitHub repository:https://github.com/MYZY-AI/Muyan-TTS
• HuggingFace model library:https://huggingface.co/MYZY-AI/Muyan-TTS
• arXiv technical paper:https://arxiv.org/pdf/2504.19146

Application scenarios of Muyan-TTS

• Podcasts and audiobooksGenerates long, coherent content while saving recording time.
• Video dubbingQuickly synthesize English scripts and voiceovers, adaptable to different characters.
• AI characters and voice assistantsGenerate distinctive character voices and provide a natural interactive experience.
• News BroadcastIt efficiently converts text to speech, making it suitable for broadcasting on smart devices.
• Education and GamesGenerate instructional audio and game narration to enhance the learning and entertainment experience.