DiaMoE-TTS - A multi-dialect TTS framework open-sourced by Tsinghua University and Giant Interactive.

What is DiaMoE-TTS?

DiaMoE-TTS is a multi-dialect speech synthesis (TTS) framework jointly developed by Tsinghua University and Giant Interactive. Based on the International Phonetic Alphabet (IPA) unified input system, the framework combines a dialect-aware Mixture-of-Experts (MoE) architecture and a low-resource adaptation strategy (PEFT) to achieve low-cost, low-barrier multi-dialect speech synthesis. The framework supports multiple dialects and less commonly spoken languages, enabling rapid modeling with limited data, and combining efficiency and flexibility. DiaMoE-TTS data, code, and methods are fully open-source, promoting dialect preservation and cultural transmission, and allowing minority languages to be heard in the digital world.

Main functions of DiaMoE-TTS

• Multi-dialect speech synthesisIt supports speech synthesis for multiple dialects and minority languages, including Cantonese, Minnan dialect, Wu dialect, etc., and can be extended to special types such as Peking Opera rhyming dialogue, allowing niche languages to speak.
• Low resource adaptationThrough PEFT strategy and data augmentation, it can quickly adapt to new dialects and synthesize natural and fluent speech under the condition of low resource conditions with only a few hours of corpus.
• High scalabilityIt is open source across the entire chain, providing complete data preprocessing, training, and inference code, supporting multiple languages, and facilitating reproduction and extension by researchers and developers.
• Highly natural speech synthesisThe dialect-aware MoE architecture uses a dynamic gating mechanism to select expert networks, preserving the unique timbre and rhythm of each dialect and improving the naturalness of speech.

The technical principle of DiaMoE-TTS

• Unified IPA frontendUsing the International Phonetic Alphabet (IPA) as a unified input system, the speech of all dialects is mapped to the same phoneme space, eliminating differences between dialects and ensuring the consistency and generalization ability of model training.
• Dialect-aware MoE architectureMultiple expert networks are introduced, with each expert focusing on learning the features of one or several dialects, avoiding the "style averaging" problem of a single network. The most suitable expert network is automatically selected for speech synthesis based on the input IPA, while dialect classification auxiliary loss is added to enhance the discriminative ability of the expert network.
• Low-Resource Adaptation Strategy (PEFT)The Conditioning Adapter and LoRA are integrated into the text embedding and attention layers, requiring only minor parameter adjustments to achieve dialect expansion, while the backbone network and MoE module remain frozen. Techniques such as pitch perturbation and speech rate perturbation are used to improve the model's synthesis performance under low-resource conditions.
• Multi-stage training methodBuilding upon the original F5-TTS checkpoint, IPA phoneme conversion data is introduced for pre-training to achieve smooth transfer of input formats. Joint modeling is performed using multiple open-source dialect datasets to activate the MoE structure, learn shared features, and distinguish pronunciation patterns across different dialects. Dynamic gating and dialect classification-assisted loss are used to further optimize the MoE's segmentation effect, capturing the unique features of each dialect. For new dialects with only a few hours of corpus data, the PEFT strategy combined with data augmentation is used to achieve efficient transfer while preserving existing knowledge.

DiaMoE-TTS project address

• GitHub repositoryhttps://github.com/GiantAILab/DiaMoE-TTS
• HuggingFace model libraryhttps://huggingface.co/RICHARD12369/DiaMoE_TTS
• arXiv technical paper: https://www.arxiv.org/pdf/2509.22727

Application scenarios of DiaMoE-TTS

• EducationIt provides vivid speech synthesis tools for teaching dialects and less commonly taught languages, helping students better learn and master the pronunciation of different languages.
• Cultural preservationIt helps protect and preserve dialects and less commonly spoken languages by using speech synthesis technology to record and reproduce dialects on the verge of disappearing, thus preserving cultural diversity.
• Virtual Humans and Digital ContentIt generates diverse dialect voices for virtual humans and digital assistants, enriching the expressiveness of virtual characters and enhancing the user experience.
• Digital Culture and TourismProviding audio guides in multiple dialects at tourist attractions enhances tourists' sense of identity and affinity with the local culture.
• Cross-border exchangesIt supports speech synthesis in multiple languages and dialects, promoting communication and understanding between people from different language backgrounds.