AB
AB
AiBoss

All Categories

project

LongCat-AudioDiT - Meituan's open-source text-to-speech model

LongCat-AudioDiT is a high-fidelity diffusion-based text-to-speech (TTS) model open-sourced by Meituan. Its core innovation lies in directly generating the text through diffusion in the waveform latent space, rather than using intermediate representations like Mel-frequency spectra, effectively avoiding error accumulation...

What is LongCat-AudioDiT?

LongCat-AudioDiT is a high-fidelity diffusion-based text-to-speech (TTS) model open-sourced by Meituan. Its core innovation lies in direct diffusion generation within the waveform latent space, rather than traditional intermediate representations like Mel-spectrum, effectively avoiding error accumulation and significantly simplifying the process. The model introduces Adaptive Projection Guidance (APG) to replace traditional CFG, correcting the training-inference mismatch problem. In the Seed benchmark, version 3.5B surpasses the previous state-of-the-art (Seed-TTS 0.809) with a Chinese similarity of 0.818, achieving top-tier zero-shot speech cloning performance. The model is open-source under the MIT license, providing both 1B and 3.5B versions and complete inference tools.

Main functions of LongCat-AudioDiT

  • Text-to-speechConverts any text into 24kHz high-fidelity natural speech, supporting multi-language synthesis including Chinese and English.
  • Zero-sample speech cloningWith just 3-10 seconds of reference audio, you can accurately replicate the voice of any speaker without any special training.
  • Waveform latent space generationThe model diffuses directly in the waveform latent space, avoiding information loss in traditional Mel spectrum and simplifying the pipeline.
  • Adaptive projection guidanceThe model uses APG technology to replace traditional CFG, correcting the training-inference mismatch and improving generation quality.
  • Flexible Inference InterfaceIt provides both CLI command line and Python API modes, supporting single-line synthesis and batch processing.
  • Dual-specification modelThe open-source 1B lightweight version and the 3.5B high-quality version meet the speed and quality requirements of different scenarios.

How to use LongCat-AudioDiT

  • Environmental preparation: Clone repository execution pip install -r requirements.txt Install dependencies.
  • Loading Model:pass AudioDiTModel.from_pretrained("meituan-longcat/LongCat-AudioDiT-1B") Load the pre-trained model onto the GPU.
  • Prepare textUse AutoTokenizer to encode the target text into the model input format.
  • Setting parameters: Specifies the generation duration, diffusion steps, guidance method (CFG or APG), and intensity.
  • Execution reasoning: Call the model to generate audio, TTS directly input text, clone requires additional reference audio and prompt text.
  • Save resultsUse soundfile to write the generated waveform data to a WAV file to complete the output.

Key information and usage requirements for LongCat-AudioDiT

  • DeveloperMeituan
  • technical routeDiffusion model + direct generation of waveform latent space
  • Model size: 1B (Lightweight Edition), 3.5B (Flagship Edition)
  • Audio quality24kHz sampling rate
  • Core InnovationWaveform latent space diffusion, adaptive projection guidance (APG)
  • performance levelSeed benchmark, state-of-the-art (SOTA), Chinese similarity 0.818
  • hardwareNVIDIA GPU (CUDA supported), ≥8GB VRAM recommended.
  • software：Python 3.8+, PyTorch, transformers, librosa
  • Dependency installation:implement pip install -r requirements.txt

LongCat-AudioDiT's core advantages

  • End-to-end simplificationIt generates directly in the waveform latent space, without the need for intermediate representations such as Mel spectrum, avoiding error accumulation and making the process simpler.
  • SOTA Voice CloningSeed achieves a benchmark Chinese similarity of 0.818, surpassing the previous state-of-the-art model Seed-TTS, and its zero-sample cloning performance is top-notch.
  • Inference quality optimizationIt supports Adaptive Projection Guidance (APG) to replace traditional CFG, correcting training-inference mismatch and generating more stable and natural results.
  • Flexible dual versionOffers a fast and lightweight 1B version and an ultra-high-quality 3.5B version, adaptable to different scenario needs.

LongCat-AudioDiT project address

  • GitHub repositoryhttps://github.com/meituan-longcat/LongCat-AudioDiT
  • HuggingFace model library:
    • https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B
    • https://huggingface.co/meituan-longcat/LongCat-AudioDiT-3.5B

Comparison of LongCat-AudioDiT with similar products

Model Developer technical route Open source situation Core features
LongCat-AudioDiT Meituan Diffusion model + waveform latent space Completely open source (MIT) Seed benchmark SOTA, APG bootstrapping, end-to-end generation
Seed-TTS ByteDance diffusion model Closed source Former SOTA, high-quality voice cloning
CosyVoice Ali Tongyi Flow Matching open source Supports command control and cross-language synthesis

Application scenarios of LongCat-AudioDiT

  • Audio content productionThe model can quickly generate audiobooks, podcasts, news broadcasts, and other content, and supports multi-character voice cloning to reduce production costs.
  • Intelligent Customer Service SystemCustomize brand-specific voices for businesses to achieve natural and fluent voice interaction services and enhance user experience.
  • Game and animation voice actingBy replicating character voices with zero samples, the dubbing costs for multilingual localization of games and animations are significantly reduced.
  • Virtual Human Live StreamingIt provides digital humans with high-fidelity real-time voice-driven capabilities, significantly enhancing the immersion and realism of virtual live streaming.
  • Accessibility toolsIt can read text aloud to visually impaired users or help aphasic individuals reconstruct their own unique voice through a small amount of recording.
Back