MOSS-TTSD - An open-source spoken dialogue speech generation model from Tsinghua University's research lab.

What is MOSS-TTSD?

MOSS-TTSD (Text to Spoken Dialogue) is an open-source spoken dialogue speech generation model developed by the Tencent AI Lab at Tsinghua University. It can convert text-based dialogue scripts into natural, fluent, and expressive dialogue speech, supporting bilingual generation in Chinese and English. The model is based on an advanced semantic-phonological neural network audio codec and a large-scale pre-trained language model, trained using over 1 million hours of single-speaker speech data and 400,000 hours of dialogue speech data. It supports zero-shot speech cloning and can generate accurate speaker switching voices based on the dialogue script, making it suitable for various scenarios such as AI podcasts, interviews, and news reports.

Main functions of MOSS-TTSD

• Highly expressive dialogue speech generationIt can convert dialogue scripts into natural and expressive dialogue speech, accurately capturing the rhythm, intonation and other characteristics of the dialogue.
• Zero-sample multi-speaker timbre cloningIt supports generating accurate speaker switching voices based on the dialogue script, achieving voice cloning for two speakers without the need for additional samples.
• Bilingual support in Chinese and EnglishIt can generate high-quality dialogue voice in both Chinese and English.
• Long-form speech generationBased on a low bit rate codec and an optimized training framework, it can generate ultra-long speech in one go, avoiding unnatural transitions when splicing speech segments.
• Fully open source and commercially readyThe model weights, inference code, and API interfaces are all open source and can be used for free in commercial applications.

The technical principle of MOSS-TTSD

• Basic model architectureMOSS-TTSD is based on the Qwen3-1.7B-base model and employs a discretized speech sequence modeling method. The model discretizes the speech using an eight-layer RVQ (Residual Vector Quantization) codebook, converting continuous speech signals into discrete token sequences. These token sequences are generated through autoregression and a delay pattern, and finally, the tokenizer's decoder reconstructs the speech from the tokens.
• Speech Discretization and Encoder InnovationOne of its core innovations is the XY-Tokenizer, a specially designed speech discretization encoder. The XY-Tokenizer is trained using a two-stage, multi-task learning approach:
  • Phase 1Training automatic speech recognition (ASR) and reconstruction tasks allows the encoder to retain coarse-grained acoustic information while encoding semantic information.
  • Phase TwoWith a fixed encoder and quantization layer, only the decoder is trained, supplementing fine-grained acoustic information through reconstruction loss and GAN loss. XY-Tokenizer can model both semantic and acoustic information simultaneously at a bit rate of 1kbps and a frame rate of 12.5Hz, outperforming other similar codecs.
• Data processing and pre-trainingMOSS-TTSD was trained using approximately 1 million hours of single-speaker speech data and 400,000 hours of dialogue speech data. The team designed an efficient data processing pipeline to sift out high-quality single-speaker and multi-person dialogue speech from massive amounts of raw audio and annotate it. The model was pre-trained using TTS data, and the use of 1.1 million hours of Chinese and English TTS data significantly enhanced the prosody and expressiveness of the speech.
• Long speech generation capabilityBased on an ultra-low bit rate codec, MOSS-TTSD supports audio generation up to 960 seconds long, enabling the generation of ultra-long speech segments in one go and avoiding unnatural transitions between spliced speech fragments.

MOSS-TTSD project address

• Project official websitehttps://www.open-moss.com/en/moss-ttsd/
• Github repositoryhttps://github.com/OpenMOSS/MOSS-TTSD
• HuggingFace model libraryhttps://huggingface.co/fnlp/MOSS-TTSD-v0.5
• Experience the demo onlinehttps://huggingface.co/spaces/fnlp/MOSS-TTSD

Application scenarios of MOSS-TTSD

• AI podcast productionMOSS-TTSD generates natural and fluent conversational speech, making it particularly suitable for producing AI podcasts. It can simulate realistic conversational scenarios and generate high-quality podcast content.
• Film and television dubbingThe model supports high-expressive dialogue speech generation in both Chinese and English, and can perform zero-sample timbre cloning, making it suitable for dialogue dubbing in film and television works.
• Long interviewMOSS-TTSD supports audio generation up to 960 seconds long, enabling the generation of ultra-long audio clips in one go. This avoids unnatural transitions between audio segments, making it ideal for generating audio for long interviews.
• News reportIn news reporting, MOSS-TTSD can generate natural, conversational voices to deliver news content and enhance its appeal.
• e-commerce live streamingThe model can be used in e-commerce live streaming scenarios such as digital human dialogue to attract viewers by generating natural conversational voices.