AB
AB
AiBoss

All Categories

project

MegaTTS 3 - A zero-shot speech synthesis system developed by ByteDance in collaboration with Zhejiang University

MegaTTS 3 is a zero-shot text-to-speech synthesis system developed by ByteDance in collaboration with Zhejiang University. It employs a lightweight diffusion model with only 0.45 bytes of parameters, enabling efficient generation of high-quality speech. The system decomposes speech into content, audio, and...

What is MegaTTS 3?

MegaTTS 3 is a zero-shot text-to-speech synthesis system developed by ByteDance in collaboration with Zhejiang University. It employs a lightweight diffusion model with only 0.45 bytes of parameters, enabling efficient generation of high-quality speech. The system decomposes speech into content, timbre, prosody, and other attributes, modeling them separately. It supports Chinese, English, and mixed Chinese-English speech synthesis, and possesses ultra-high-quality speech cloning capabilities, mimicking a target voice from just a few seconds of audio samples. It also supports controllable features such as accent intensity control. MegaTTS 3 can be applied to various scenarios including speech synthesis, speech editing, and cross-language speech synthesis.

Main features of MegaTTS 3

  • Zero-sample synthesisIt can generate the speech of the target speaker with minimal prompts, without requiring specific speech data of the target speaker, thus enabling rapid speech cloning.
  • Multilingual supportIt supports Chinese, English, and mixed Chinese-English speech synthesis to meet the needs of different language scenarios.
  • High-quality audio outputThe generated speech is natural and fluent, with clear sound quality and a high degree of similarity to the target speaker.
  • tone controlIt allows you to adjust the timbre of the generated speech to make it closer to the target speaker or add specific timbre effects.
  • Rhythm AdjustmentIt supports control over the rhythm of speech, such as speech rate and intonation, making speech more expressive.
  • Accent intensity controlBy adjusting the parameters, speech with different accent intensities can be generated to simulate various language styles.
  • Rapid CloningIt can quickly generate the speech of the target speaker with just a few seconds of audio sample, achieving efficient speech cloning.

The technical principles of MegaTTS 3

  • Lightweight diffusion modelMegaTTS 3 employs a lightweight diffusion model (TTS Diffusion Transformer) with only 0.45B parameters, ensuring high efficiency while generating high-quality speech. The diffusion model generates target speech by progressively adding and removing noise, and its core consists of a forward process (adding noise) and a backward process (denoising). The backward process is used to generate data samples.
  • Speech decomposition and modelingMegaTTS 3 decomposes speech into different attributes such as content, timbre, prosody, and phase, and designs appropriate modules for modeling each attribute.
    • Timbre ModelingUse global vectors to model timbre because timbre is a global property that changes slowly over time.
    • Prosody ModelingWe use a latent code language model to fit the distribution of prosody because prosody changes rapidly in sentences, and the language model can capture both local and long-range dependencies.
    • Content Modeling: A VQGAN-based acoustic model is used to generate spectrograms.
    • Phase modelingThe phase is appropriately constructed by a GAN-based vocoder, and no language model is required to model the phase.
  • Data and TrainingMegaTTS 3 is trained on a large-scale, multi-domain dataset containing 20,000 hours of speech data. This enables the model to perform exceptionally well on zero-shot speech synthesis, speech editing, and cross-language speech synthesis tasks.
  • Sparse alignment algorithmMegaTTS 3 introduces a sparse alignment algorithm that provides sparse alignment boundaries to guide the latent diffusion transform (DiT), reducing alignment difficulty without shrinking the search space and achieving high naturalness.

MegaTTS 3 project address

Application scenarios of MegaTTS 3

  • academic researchResearchers can use it to test speech synthesis technology and analyze the effects of latents.
  • Educational SupportConvert textbooks into audio to create audiobooks and enhance the learning experience.
  • Content creationGenerate narration for videos or podcasts, saving on manual recording costs.
  • Voice interactionDevelopers can integrate it into devices to enable Chinese and English voice conversations.
Back