Voxtral TTS - Mistral AI's open-source text-to-speech model

What is Voxtral TTS?

Voxtral TTS is an open-source text-to-speech model from Mistral AI, based on a 4 billion parameter architecture and supporting 9 languages. The model boasts an ultra-low latency of 90 milliseconds and a 6x faster real-time generation speed, achieving zero-sample speech cloning with only 3-5 seconds of audio. The model can be deployed on edge devices, requiring only 3GB of memory after quantization, and its API is priced at $0.016 per thousand characters. Voxtral TTS completes Mistral's end-to-end voice AI platform, suitable for scenarios such as voice customer service, real-time translation, and audiobooks, and outperforms competitor ElevenLabs in multiple evaluations.

Main functions of Voxtral TTS

• Multilingual speech synthesisIt supports text-to-speech generation in nine languages: English, French, German, Spanish, Dutch, Portuguese, Italian, Hindi, and Arabic.
• Zero-sample speech cloningIt only requires 3-5 seconds of reference audio to clone the voice of any speaker and supports cross-language timbre transfer.
• Emotional style controlThe model can adjust the emotional state (such as anger, happiness, sadness) and parameters such as speech rate, tone, and volume of the generated speech.
• Ultra-low latency real-time generationThe initial audio time is only 90 milliseconds, and the real-time factor is 6 times that of real-time dialogue scenarios.
• End-side equipment deploymentThe model can run on edge devices such as smartwatches and mobile phones, and requires only about 3GB of memory after quantization.

Voxtral TTS Technical Principles

• Three-module cascaded architectureThe model consists of a Transformer language model with 3.4B parameters, a stream-matching acoustic model with 390M parameters, and a neural audio codec with 300M parameters, with a total of approximately 4 billion parameters.
• Text to Discrete RepresentationBased on the Ministral 3B backbone network, a streaming BERT-style masked language model is used to convert the input text into discrete speech tokens.
• Stream matching spectrum generationThe model uses flow matching technology to replace the traditional diffusion model, which quickly converts discrete tokens into continuous Mel spectrograms, achieving faster inference speed.
• Neural audio encoding and decodingThe spectrogram is reconstructed into a high-quality audio waveform using a 300M parameter codec, ensuring the naturalness of the output.
• Edge-optimized deploymentSupports INT8/INT4 quantization compression, reducing model size to 3GB memory usage, and adapting to running on edge devices such as smartphones.

Key information and usage requirements of Voxtral TTS

• Release timeOfficially released by Mistral AI on March 26, 2026.
• Model sizeThe total number of parameters is approximately 4 billion, including a 3.4B Transformer language model, a 390M stream-matching acoustic model, and a 300M neural audio codec.
• Supported languagesThe languages are: English, French, German, Spanish, Dutch, Portuguese, Italian, Hindi, and Arabic, totaling nine languages.
• Performance indicatorsThe initial audio time is 90 milliseconds, the real-time factor is 6 times, and voice cloning only requires 3-5 seconds of reference audio.
• LicenseOpen source weights are licensed under the Creative Commons license, and the API is priced at $0.016 per thousand characters.
• Hardware RequirementsLocal deployment requires at least 3GB of memory (quantized version), and supports edge devices such as smartwatches, smartphones, and laptops.

Voxtral TTS's core advantages

• Open source and customizableThe model weights are completely open source, allowing enterprises to deploy them locally and fine-tune them as needed, thus avoiding the data privacy risks associated with relying on third-party cloud services.
• Ultra-low latency and high performanceThe initial audio time is only 90 milliseconds, and the real-time factor is 6 times higher, which is significantly better than similar competing products and meets the needs of real-time dialogue scenarios.
• End-side deployment capabilityAfter quantization, it only requires 3GB of memory and can run on edge devices such as smartphones and smartwatches without cloud connectivity.
• Zero-sample speech cloningIt can clone the voice of any speaker with only 3-5 seconds of audio, supports cross-language timbre transfer, and greatly reduces the cost of voice customization.

How to use Voxtral TTS

• Online experienceTo generate audio, simply access the Mistral Studio console or the Le Chat platform, enter text, and select the voice parameters.
• API callsRegister a Mistral account to obtain an API key, send text and an optional reference audio URL via the REST API, and receive the generated audio file.
• Open source local deploymentDownload model weights from Hugging Face, load them using the PyTorch or transformers library, and run inference on your local GPU or CPU.

Voxtral TTS project address

• Project official websitehttps://mistral.ai/news/voxtral-tts
• HuggingFace model libraryhttps://huggingface.co/mistralai/Voxtral-4B-TTS-2603
• Technical Papers: https://mistral.ai/static/research/voxtral-tts.pdf

Comparison of Voxtral TTS with similar competitors

Application scenarios of Voxtral TTS

• Real-time voice interactionThe model supports the construction of low-latency intelligent customer service, voice assistants, and chatbots, achieving natural human-computer dialogue with a 90-millisecond response time.
• Cross-language content localizationIt translates video, podcast, and other content into nine target languages while preserving the original speaker's vocal characteristics, thus reducing the cost of multilingual production.
• Personalized audio contentThe model can clone specific voices to generate audiobooks, news broadcasts, and educational training audio, meeting the customized needs of brands.
• Immersive entertainment experienceProvides emotionally controllable dynamic voices for game NPCs and interactive narratives, enhancing player immersion.
• Accessibility toolsIt can read text messages aloud to visually impaired users or reconstruct personalized digital voices for people with speech impairments.