OmniVoice - An open-source multilingual TTS model from the Xiaomi team

What is OmniVoice?

OmniVoice is a large-scale, multilingual TTS model open-sourced by the next-generation Kaldi team at Xiaomi AI Labs. It supports zero-shot speech cloning for over 600 languages. The model employs a minimalist non-autoregressive diffusion architecture, combined with full-codebook randomized masking and LLM initialization techniques, achieving 40 times faster real-time inference speed. OmniVoice supports timbre design, audio denoising, sub-language control (such as laughter), and pronunciation correction, achieving state-of-the-art (SOTA) levels in sound quality and language coverage, and exhibiting particularly strong generalization capabilities for low-resource languages.

OmniVoice's main functions

• Large-scale multilingual synthesisSupports zero-shot TTS for 600+ languages, covering languages from high-frequency to low-resource languages, and is trained on 580,000 hours of open-source data.
• Zero-sample speech cloningIt can clone any speaker's voice with just 3-10 seconds of reference audio, and supports automatic transcription (built-in Whisper) or manually provided text.
• Attributed tone designIt generates customized voices directly from natural language descriptions (gender, age, tone, dialect/accent, whisper style, etc.) without the need for reference audio.
• Reference audio noise reductionIt can process reference audio with noise or reverberation, extract clean speaker features, and avoid synthesized speech carrying environmental noise.
• Secondary language control:insert [laughter],[sigh] Add emotions such as laughter and sighs to tags.
• Pronunciation correctionUse Pinyin (e.g.) ZHE2) or CMU phonemes (such as [B EY1 S]Correcting the pronunciation of polyphonic characters and proper nouns

How to use OmniVoice

• Installation and DeploymentUsers can deploy OmniVoice by using the pip command or by cloning the source code from GitHub and executing the installation command.
• Quick ExperienceUsers can quickly experience OmniVoice's speech synthesis effects without writing any code by starting a local web service or accessing the HuggingFace online demo.
• Voice cloningDevelopers can use the Python API to load pre-trained models and implement zero-shot speech cloning by providing reference audio and transcribed text.
• Tone DesignUsers can directly design and generate specific timbres by describing the speaker's gender, age, tone, accent, and other attributes using natural language.
• command line toolsThe system supports single-speech synthesis or batch inference tasks across multiple GPUs via command-line tools.
• Fine-grained controlUsers can insert specific tags into the synthesized text to add paralinguistic expressions such as laughter and sighs, or use pinyin and phoneme symbols to correct the pronunciation of polyphonic characters and proper nouns.

Key information and usage requirements for OmniVoice

• Technical positioningOmniVoice is an open-source, non-autoregressive TTS system based on a diffusion language model developed by Xiaomi AI Labs. It adopts a minimalist bidirectional Transformer architecture to directly map text to multiple codebook acoustic tokens, abandoning the traditional cascaded pipeline, and supports zero-sample speech cloning and synthesis for 600+ languages.
• Core parametersThe model has 0.8B parameters and is initialized based on Qwen3-0.6B. It was trained on 581,000 hours of open-source speech data covering 646 languages, and its inference speed reaches RTF 0.025 (40 times real-time). In Chinese, English and multilingual benchmark tests, the speaker similarity and intelligibility metrics have reached the state-of-the-art level.
• Functional boundariesSupports zero-sample speech cloning of 3-10 second reference audio (including noise reduction of noisy/reverberant audio), timbre design based on attributes such as gender/age/pitch/dialect/accent, and paralinguistic control and pronunciation correction through tags and pinyin/phonemes.
• Hardware environmentRequires an NVIDIA GPU that supports CUDA (H800/H20 recommended for best performance) or an Apple Silicon device that supports MPS to run model inference.
• Software DependencyYou need to install PyTorch (matching the CUDA version) in a Python virtual environment and install the omnivoice package via pip or source code to meet runtime dependencies.

OmniVoice's core advantages

• Minimalist and efficient architectureIt adopts a single-stage non-autoregressive diffusion language model architecture, using a bidirectional Transformer to directly map text to multiple codebook acoustic tokens, eliminating error propagation and information bottlenecks in traditional cascaded pipelines, and achieving a dual breakthrough in architectural simplicity and performance.
• Widest language coverageTrained on 581,000 hours of open-source data, it supports zero-shot speech synthesis for 600+ languages and maintains an extremely low word error rate even for low-resource, less common languages. It is currently the TTS model with the widest language coverage.
• Extremely fast reasoning speedBy employing a full-codebook random masking strategy and an efficient architecture design, an inference speed of RTF as low as 0.025 is achieved, which is 40 times faster than real-time and significantly outperforms autoregressive models.
• Sound quality and intelligibility: SOTABy leveraging LLM to initialize and inherit pre-trained language knowledge, speaker similarity (SIM-o) and intelligibility (WER) outperform existing non-autoregressive models and commercial systems in benchmark tests such as LibriSpeech-PC and Seed-TTS.
• Multidimensional controllabilityIt supports zero-sample speech cloning, attribute-based timbre design, noise reduction with noise frequency, insertion of secondary speech symbols (laughter/sigh), and pinyin/phoneme-level pronunciation correction, meeting the fine-grained control requirements in complex scenarios.

OmniVoice project address

• GitHub repositoryhttps://github.com/k2-fsa/OmniVoice
• HuggingFace model libraryhttps://huggingface.co/k2-fsa/OmniVoice
• arXiv technical paper: https://arxiv.org/pdf/2604.00688
• Experience the demo onlinehttps://huggingface.co/spaces/k2-fsa/OmniVoice

Comparison of OmniVoice with similar competing products

Application scenarios of OmniVoice

• Multilingual content localizationCreators can use OmniVoice's support for 600+ languages to quickly convert podcasts, audiobooks, educational courses, and other content into different language versions, while maintaining the consistency of the original speaker's timbre through zero-sample cloning.
• Game and film dubbingGame developers and production companies can generate diverse voices for characters by referencing audio cloning or designing attributed voices.
• Intelligent Customer Service and AssistantEnterprises can deploy OmniVoice to build a multilingual intelligent customer service system, clone the voice of a specific brand ambassador with zero samples, or generate a custom voice assistant that matches the brand image through voice design.
• Education and Language LearningEducational institutions can develop rare language learning materials that help learners master the accurate pronunciation of Chinese polyphonic characters or English proper nouns through pinyin/phoneme-level pronunciation correction functions.