AB
AB
AiBoss

All Categories

project

Kyutai TTS - Streaming text-to-speech technology from Kyutai Labs

Kyutai TTS is a streaming text-to-speech (TTS) technology developed by Kyutai Labs, a French artificial intelligence research institution. It's an innovative speech synthesis system that can convert text into natural, fluent speech in real time, without waiting for complete text output...

What is Kyutai TTS?

Kyutai TTS is a streaming text-to-speech (TTS) technology developed by Kyutai Labs, a French artificial intelligence research institution. It's an innovative speech synthesis system that can convert text into natural, fluent speech in real time, starting audio generation without waiting for complete text input, with extremely low latency (only 220 milliseconds). It supports streaming text transmission and performs exceptionally well in real-time interactive scenarios, such as intelligent customer service, real-time translation, and live streaming. It supports English and French and features voice cloning, matching a speaker's timbre and intonation using a 10-second audio sample. Kyutai TTS supports long text generation, breaking through the time limitations of traditional TTS systems, making it suitable for scenarios such as news broadcasting and audiobooks.

Main functions of Kyutai TTS

  • Streaming text transmissionIt supports text streaming, allowing audio generation to begin without requiring complete text, making it suitable for real-time interactive scenarios such as intelligent customer service, real-time translation, and live streaming.
  • low latencyWith a single NVIDIA L40S GPU, Kyutai TTS can handle 32 requests simultaneously with a latency of only 350 milliseconds, enabling it to quickly respond to a large number of user requests.
  • High-fidelity soundIt supports voice cloning using 10-second audio samples, generating natural and fluent speech with speaker similarity of 77.1% (English) and 78.7% (French), and word error rate (WER) of 2.82% and 3.29%, respectively.
  • Long text generationBreaking through the traditional 30-second limit of TTS systems, it can handle long articles and is suitable for scenarios such as news broadcasting and audiobooks.
  • Multilingual supportCurrently supports English and French.

Kyutai TTS Technical Principles

  • Delayed Flow Modeling (DSM)DSM is the core architecture of Kyutai TTS, treating speech and text as two time-aligned data streams. The text stream is delayed by several time frames relative to the audio stream, allowing the model to "see the speech a little in the future," improving the accuracy and naturalness of the generated speech. During inference, the model progresses step by step in time, without waiting for the complete audio input, enabling streaming generation.
  • audio codecThe model uses a custom causal audio codec (such as Mimi) to encode speech into low-frame-rate discrete tokens, supporting real-time streaming. This enables the model to achieve efficient real-time generation while maintaining high-quality speech output.
  • High concurrency and low latencyKyutai TTS can handle 32 requests simultaneously on a single NVIDIA L40S GPU with a latency of only 350 milliseconds.
  • Voice cloning and personalizationThe model supports sound cloning using 10-second audio samples, matching the pitch, intonation, tone, and recording quality of the original audio.
  • Word timestampKyutai TTS generates speech with precise timestamps for each word, enabling real-time captioning and interactive applications.

Kyutai TTS project address

  • Project official websitehttps://kyutai.org/next/tts

Application scenarios of Kyutai TTS

  • Intelligent Customer ServiceKyutai TTS's low latency feature allows the system to generate an instant voice response when a user asks a question in an intelligent customer service scenario, without waiting for the user to finish speaking, greatly improving interaction efficiency and user experience.
  • Real-time translationIn scenarios such as cross-border business negotiations and international academic exchanges, Kyutai TTS can quickly convert translated text into speech, enabling seamless communication.
  • Video conferencing and live streamingKyutai TTS provides real-time caption generation for video conferencing and live streaming. It can quickly and accurately generate synchronized captions, making it easier for viewers to understand the content.
  • EducationKyutai TTS provides high-quality text-to-speech services for visually impaired individuals, helping them better access information. It can also be used in online education platforms to provide students with engaging teaching content and enhance their learning experience.
  • Media ProductionKyutai TTS can handle the speech generation of long articles and is suitable for scenarios such as news broadcasting and audiobook production.
  • Voice navigationKyutai TTS's high concurrency processing capabilities support scenarios such as in-vehicle navigation and public transportation voice prompts, providing users with clear and timely voice broadcasts.
Back