dots.tts - A speech synthesis foundation model jointly open-sourced by Xiaohongshu and Shanghai Jiao Tong University.

What is dots.tts?

dots.tts is a 2 billion-parameter fully continuous autoregressive speech synthesis foundation model jointly open-sourced by the Xiaohongshu dots team and the X-LANCE Lab at Shanghai Jiao Tong University. The model directly generates 48kHz audio block-by-block in a continuous latent space, achieving state-of-the-art timbre similarity and content accuracy on benchmarks such as Seed-TTS-Eval. It supports zero-shot cloning, streaming output, and low-latency full-duplex dialogue. The project includes full open-source code and weights, and extends the precise speech editing capabilities of dots.tts.edit.

The main functions of dots.tts

• Zero-sample timbre cloningWith only 3–10 seconds of reference audio, the voice can be replicated to read any new text. It supports cross-language cloning and achieves state-of-the-art (SOTA) performance in content accuracy and speaker similarity on the Seed-TTS-Eval benchmark.
• Text-to-speech synthesisIt provides high-quality speech generation from random timbre sampling to a specified speaker, directly outputting 48 kHz audio based on a 2B parameter continuous autoregressive model.
• Streaming generation and low-latency interactionIt natively supports streaming output, and audio can be generated block by block after the text prefix is input; in 1T1A dual-stream mode, the voice side responds immediately for each text token output by the upstream LLM, and the audio first packet latency is as low as 54.4 milliseconds, which is suitable for real-time voice agents and full-duplex dialogue.
• Precise speech editingIt supports text replacement, mood adjustment, pitch and speed modification, pause insertion or deletion for existing recordings, and supports multiple operations to be completed in one command at a time, while the unedited area remains unchanged.
• Multilingual supportIt covers 24 languages and achieves an average speaker similarity of 83.9 in the MiniMax multilingual evaluation, with 19 languages achieving first place in individual categories.
• Multiple inference speedsIt offers multiple checkpoints, including Base, SOAR, MeanFlow (4 steps), two-step sCM, and single-step DMD, allowing users to choose between sound quality and inference speed as needed.

The technical principles of dots.tts

• Fully Continuous Autoregressive ArchitectureThe dots.tts approach abandons the traditional method of quantizing speech into discrete tokens. The entire generation chain runs entirely in a continuous latent space: the model predicts acoustic segments block by block in an autoregressive manner, and restores them to 48 kHz waveforms through a BigVGAN-style decoder, fundamentally avoiding the information bottleneck and loss of timbre details caused by discrete quantization.
• Semantic continuous representationThe model uses AudioVAE based on HoliTok to compress the original waveform into a continuous latent variable of 25 FPS. The encoder achieves a balance between reconstruction fidelity and latent space structuring through semantic training, so that the similarity of the reconstructed speaker reaches 0.969, which preserves the upper limit of rich timbre and acoustic details for subsequent autoregressive generation.
• Error control mechanismTo address the inherent problem of small deviations in previous steps accumulating and spreading in continuous autoregression, dots.tts introduces a causal semantic encoder to compress the generated VAE patch into a compact semantic history in real time and feed it back into the large language model. By stripping away high-variance local acoustic variations and retaining only long-range semantic summaries, it effectively suppresses error propagation, howling, and timbre drift in the generation of long sequences.
• Generation processThe generation process is centered on a large language model, which is initialized based on Qwen2.5-1.5B, directly consumes BPE text tokens and outputs the hidden state at each step; then the autoregressive stream matching head (18-layer DiT) performs denoising generation on the next acoustic patch under the conditions of LLM hidden state and autoregressive prefix, while using the global x-vector provided by the frozen CAM++ speaker encoder to ensure timbre consistency.
• Post-training accelerationThe team used SOAR self-correcting alignment to enable the model to simulate deviations in real inference during training and learn to self-correct, improving stability without rewarding the model; they used MeanFlow distillation to compress the teacher's multi-step trajectory into an average speed, achieving high-quality generation in 4 steps; they further compressed it to 2 steps using a simplified consistency model (sCM); and they obtained a natural and clear high-quality single-step model by using reward-aware distribution matching distillation (DMD) in conjunction with a dual-timescale update strategy.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use dots.tts

• Install:implement pip install dots.tts You can install it directly, or clone the source code from GitHub and then... pip install -e . For local development and installation, SGLang Omni can be additionally deployed in high-concurrency scenarios to obtain CUDA Graph acceleration and continuous batch processing support.
• Command line synthesis:use dots.tts Entry and specify --model-name-or-path、--text and target --prompt-audio and --prompt-textThis allows for zero-sample timbre cloning; omit --prompt-text It then degenerates into an x-vector clone, omitting... --prompt-audio Then random timbre sampling is performed.
• Command line editing:use dots.tts.edit Entrance, providing --source-audio and XML style --instructionThis allows for precise editing of existing recordings, such as text replacement, mood or rhythm modification.
• Python API Basic Calls:pass DotsTtsRuntime.from_pretrained() Call after loading the model generate()By passing in text and optional reference audio, you can obtain a synthesized audio tensor and save it as a file.
• Python API Streaming Output: call generate_stream() Audio tensors can be obtained block by block and pushed to the player or WebSocket in real time, achieving a low-latency experience of generating and playing simultaneously.
• Python API Two-Stream Dialogue:use DotsTtsRuntimeDoubleStreaming After the session is started, text is pushed one token at a time, and the voice side immediately begins to generate audio. The latency of the first audio packet can be as low as 54.4 milliseconds, which is suitable for real-time voice agents.
• Fine-tuning:run accelerate launch scripts/train_dots_tts.py By configuring its own data path, users can make fine-tuning adjustments for timbre or domain adaptation based on public checkpoints.
• Distillation acceleration:pass scripts/train_dots_tts_meanflow.py Using SOAR checkpoints for MeanFlow distillation of teachers can yield high-quality student models in 4-step, 2-step, or 1-step processes to accommodate different latency and concurrency requirements.
• Web Interface:run python apps/gradio/app.py Launch the visual composition interface, or run... apps/edit_playground/app.py Launch Edit Playground, upload audio in your browser, mark the editing area, and preview the results in real time.

The core advantages of dots.tts

• Modeling of fully continuous hidden spacesAbandoning discrete acoustic tokens, it directly generates them autoregressively in the continuous latent space, preserving rich timbre details such as spectral texture and breathy sounds, and reconstructing speaker similarity as high as 0.969.
• SOTA Synthesis QualityAchieving state-of-the-art (SOTA) content accuracy and timbre similarity in both Seed-TTS-Eval benchmarks, and achieving an average SIM of 83.9 in MiniMax tests for 24 languages, its overall performance surpasses mainstream solutions.
• Native streaming low latencyThe 1T1A dual-stream mode audio first packet is as low as 54.4 milliseconds, and with the SGLang Omni single-card H100, the concurrent 16-channel throughput reaches 4.76 req/s, meeting the needs of real-time voice agents.
• Unified editing capabilitiesdots.tts.edit, based on the same platform, supports precise editing of text, mood, rhythm, and pauses using XML structured instructions, and outperforms other open-source systems in the doteBench benchmark.

The project address for dots.tts

• GitHub repository: https://github.com/studio-dots-ai/dots.tts
• arXiv technical paper: https://arxiv.org/pdf/2608.02673

Comparison of dots.tts with similar competitors

Application scenarios of dots.tts

• Real-time voice agentThe 1T1A dual-stream mode has a first packet latency as low as 54.4 milliseconds, making it suitable for LLM-driven real-time full-duplex conversations and voice assistants.
• Zero-sample timbre cloningIt can replicate any timbre with just a few seconds of reference audio, supports cross-language cloning, and is suitable for audiobooks, voice-over, and personalized content creation.
• Multilingual content generationIt covers 24 languages and maintains high speaker similarity, providing multilingual voice broadcasting and localization services suitable for global products.
• Precise speech editingDots.tts.edit allows you to replace text, adjust mood, modify rhythm, and adjust pauses in existing recordings. It is suitable for podcast post-production and audio content editing.
• Live broadcast and real-time reportingIt can be used with LLM streaming output to generate and play content simultaneously, making it suitable for news broadcasting, live e-commerce, and real-time information broadcasting scenarios.