AB
AB
AiBoss

All Categories

project

Llasa TTS - An open-source text-to-speech model from the Hong Kong University of Science and Technology

Llasa TTS is an open-source text-to-speech (TTS) model developed by the Hong Kong University of Science and Technology based on the LLaMA architecture. It supports high-quality speech synthesis and cloning. Llasa TTS is based on a single-layer vector quantization (VQ) codec and a single Tr...

What is Llasa TTS?

Llasa TTS is an open-source text-to-speech (TTS) model developed by the Hong Kong University of Science and Technology based on the LLaMA architecture. It supports high-quality speech synthesis and cloning. Based on a single-layer vector quantization (VQ) codec and a single Transformer architecture, Llasa TTS is fully aligned with the standard LLaMA model, generating natural and fluent speech and supporting features such as emotional expression and timbre cloning. Llasa TTS performs exceptionally well in both training and inference phases, improving speech naturalness, prosodic accuracy, and emotional expression by leveraging computational resources that extend training and inference times. Llasa TTS provides models with 1B, 3B, and 8B parameters and supports multilingual synthesis.

Main functions of Llasa TTS

  • High-quality speech synthesisIt generates natural and fluent speech, supports both Chinese and English, and is suitable for a variety of application scenarios.
  • Emotional expressionInjecting emotional information generates speech with emotional overtones such as happiness, anger, and sadness, enhancing the naturalness and expressiveness of the speech.
  • Voice cloningWith only a small number of audio samples (such as 15 seconds), it can clone the timbre and emotion of a specific human voice to achieve personalized speech synthesis.
  • Long text supportIt supports processing long text input and generating coherent speech output, making it suitable for scenarios such as audiobooks and voice broadcasts.
  • Zero-shot learningIt supports speech synthesis for speakers or emotions that have not been seen before, without the need for additional fine-tuning.

Llasa TTS Technical Principles

  • Transformer-based architectureBased on a single Transformer architecture, it is fully aligned with standard large-scale language models. It uses a single-layer vector quantization (VQ) codec to convert speech waveforms into discrete speech tokens, modeled using Transformer.
  • speech segmenter:
    • codingThe speech signal is decomposed into semantic features and acoustic features, which are extracted based on pre-trained Wav2Vec2-BERT and convolutional modules, respectively.
    • QuantificationFeatures are encoded into discrete labels using an improved vector quantization (VQ) technique.
    • decodingIt decodes discrete tokens back into high-quality speech waveforms, supporting the reconstruction of semantic and acoustic information.
  • Training and Reasoning Extension:
    • Training time extensionImprove speech naturalness and prosodic accuracy by increasing model size (e.g., 1B, 3B, 8B parameters) or training data volume (e.g., 250k hours of speech data).
    • Inference time extensionIn the inference stage, a speech understanding model is introduced as a validator, and complex search strategies (such as bundle search and best candidate selection) are used to optimize the generated results, thereby enhancing emotional expression and timbre consistency.
  • Autoregressive generationBased on an autoregressive generation method, speech tags are generated one by one to ensure that the generated speech is consistent with the input text in terms of semantics and prosody.

Llasa TTS project address

Application scenarios of Llasa TTS

  • Intelligent voice assistantProvides natural and fluent voice interaction for smart devices or software, enhancing the user experience.
  • Audiobooks and Online EducationIt transforms text content into vivid speech, providing users or students with an auditory learning experience.
  • Voice broadcast and customer serviceUsed for news broadcasting, traffic information prompts, or customer service systems to provide efficient information delivery.
  • Games and EntertainmentGive game characters or virtual avatars personalized voices to enhance immersion.
  • Voice cloning and content creationCloning specific human voices for use in advertising voice-overs, video production, or personalized voice content creation.
Back