Hibiki - Kyutai Labs' real-time speech translation model

What is Hibiki?

Hibiki is an open-source decoder model from Kyutai Labs for simultaneous speech translation, capable of translating speech from one language into speech or text in another language in real time. Based on a multi-stream language model architecture, Hibiki processes source and target speech simultaneously, jointly generating text and audio tags to achieve speech-to-speech (S2ST) and speech-to-text (S2TT) translation. Hibiki uses a weakly supervised learning method, identifying the optimal word-level latency based on the perplexity of the text translation system, and training on aligned synthetic data. The Hibiki model performs exceptionally well in French-to-English translation tasks, exhibiting high translation quality, speaker fidelity, and naturalness. It supports batch translation and real-time on-device deployment, demonstrating strong practical potential.

Hibiki's main functions

• Real-time speech-to-speech translation (S2ST)It can translate speech from one language into speech in another language in real time, preserving the speaker's timbre and intonation.
• Real-time speech-to-text translation (S2TT)): Translates speech into text in the target language in real time, providing more flexible use cases.
• Low-latency translationBased on real-time accumulation of contextual information, the translation content is generated block by block with extremely low latency, approaching the level of human interpretation.
• High fidelityThe generated speech is natural and fluent, with a high degree of similarity to the timbre and intonation of the source speech, providing a user experience close to that of professional human interpreters.
• Supports batch and real-time deploymentThe inference process is simple, supports batch processing and real-time device deployment, and is suitable for large-scale applications.

Hibiki's technical principles

• Multi-stream language model architecture:
  • Synchronous processingSimultaneously receive source speech and generate target speech, and jointly model the two audio streams based on a multi-stream architecture.
  • Text and audio tagsThe model predicts the hierarchical structure of text and audio tags, enabling speech-to-text and speech-to-speech translation.
  • Causal audio codec: Encode speech into low-frame-rate discrete tokens using pre-trained causal audio codecs (such as Mimi), supporting real-time streaming.
• Weakly supervised learning and context alignment:
  • Synthetic data generationThe method involves transcribed text from translated monolingual audio and resynthesizing the target speech to generate aligned synthetic data.
  • Context Alignment: Calculate word-level alignment using the perplexity of an existing text translation system to ensure that the generated target speech is context-synchronized with the source speech.
  • Silent Insertion and Alignment-Aware TTSThe target speech is resynthesized based on the insertion of silence or using an alignment-aware TTS model to ensure that the delay of the target speech meets the requirements of real-time translation.
• Speaker similarity and classifier free guidance:
  • Speaker similarity markersThe training data is classified and labeled according to speaker similarity, which avoids data filtering and prioritizes highly similar samples during inference.
  • Classifier free guidanceAdjusting the weights of conditional labels enhances the model's control over speaker similarity, further improving speech fidelity.
• Efficient reasoning process:
  • Temperature samplingUsing temperature sampling technology combined with a causal audio codec, streaming input and output can be achieved.
  • Batch processing and real-time deploymentThe inference process is simple and efficient, supports batch processing and real-time device deployment, and is suitable for large-scale application scenarios.

Hibiki's project address

• GitHub repository:https://github.com/kyutai-labs/hibiki
• HuggingFace model library:https://huggingface.co/collections/kyutai/hibiki
• arXiv technical paper:https://arxiv.org/pdf/2502.03382

Hibiki's application scenarios

• International ConferenceReal-time translation of speeches in different languages helps participants understand the content immediately.
• Online EducationIt translates teachers' lectures in real time, facilitating barrier-free learning for students.
• TravelReal-time translation of tour guide explanations or interaction with locals enhances the travel experience.
• News interviewIt helps reporters quickly translate interview content and improves reporting efficiency.
• Customer ServiceEnables multilingual customer service communication, improving customer satisfaction.