AB
AB
AiBoss

All Categories

project

MoE-TTS - A speech synthesis framework launched by Kunlun Tech

MoE-TTS is the first role-description speech synthesis framework based on MOE launched by the Kunlun Wanwei speech team, specifically designed to improve the understanding of open-domain text descriptions. The model utilizes a Mixture-of-Experts (MOE) approach...

What is MoE-TTS?

MoE-TTS is the first MOE-based speech synthesis framework for character descriptions launched by the Kunlun Wanwei Speech Team, specifically designed to improve the understanding of open-domain text descriptions. The model combines a pre-trained Large Language Model (LLM) with speech expert modules through a Mixture-of-Experts (MoE) architecture. During training, the text module parameters are frozen, and only the speech module parameters are updated, preserving the powerful text understanding capabilities of the LLM while enhancing the accuracy of speech generation. Experiments show that MoE-TTS significantly outperforms existing commercial models in generating speech that more closely matches the descriptions, especially excelling in handling complex and open-domain descriptions.

Main functions of MoE-TTS

  • Enhanced open-domain text understandingIt can accurately understand and generate speech that matches complex, open-domain text descriptions, and supports descriptions that have not appeared in the training data.
  • Natural Language Description DriverUsers can precisely control the style and characteristics of a voice through natural language descriptions (such as "a vibrant teenage voice" or "an actor with a New York accent").
  • High-quality speech generationThe generated speech performs exceptionally well in terms of naturalness, emotional expression, and style consistency, significantly outperforming traditional TTS models.
  • Cross-modal knowledge transferThe powerful text understanding capabilities of pre-trained language models are transferred to speech generation tasks, enhancing the model's ability to understand and express complex semantics.

MoE-TTS Technical Principles

  • Pre-trained LLM as the base modelBased on a pre-trained text LLM as the base model, the parameters are frozen to retain powerful text understanding capabilities.
  • Modal routing strategyBased on the modal routing mechanism, text and speech tags are assigned to text expert and speech expert modules respectively to avoid intermodal interference.
  • Frozen Text Expert ModuleDuring training, only the parameters of the speech expert module are updated, while the parameters of the text expert module are frozen, ensuring that pre-trained knowledge is preserved during training and inference.
  • Modality-aware Transformer componentsThe core components of the Transformer layer (such as layer normalization, feedforward network, and multi-head attention) are converted into modality-aware MoE layers to further improve the model's ability to handle different modalities.
  • Speech generation moduleCombining diffusion models (such as Elucidated Diffusion Models) and VAEGAN components, discrete speech tokens are converted into high-quality continuous speech waveforms.

MoE-TTS project address

  • Technical Papershttps://teal-aquarius-c17.notion.site/MoE-TTS-Enhancing-Out-of-Domain-Text-Understanding-for-Description-based-TTS-via-Mixture-of-Experts-24e44360bf708040bff3dffe2eef805e#24e44360bf70800c9290cce2d2d14dfe

Application scenarios of MoE-TTS

  • Virtual assistants and intelligent customer serviceThis makes the voice responses of virtual assistants and intelligent customer service natural and fluent, as if they were real people, greatly improving the user experience.
  • Audio content creationIt generates high-quality voices for audiobooks, podcasts, and more, with diverse styles and rich emotions, making the content more engaging.
  • Digital Humans and Virtual Character Voice ActingPersonalized voices are generated based on character settings, instantly bringing digital humans and virtual characters to life and enhancing realism and expressiveness.
  • Education and TrainingIt supports multilingual and multi-style speech generation, helping to diversify educational content and make learning more fun and efficient.
  • Games and Interactive EntertainmentIt generates real-time voices that fit the scene, making the game highly interactive, the character dialogues vivid and realistic, and the immersive experience to the fullest.
Back