Xiaomi-MiMo-Audio - Xiaomi's open-source end-to-end large-scale voice model

What is Xiaomi-MiMo-Audio?

Xiaomi-MiMo-Audio is Xiaomi's first native end-to-end large-scale speech model, open-sourced by Xiaomi. Based on an innovative pre-training architecture and hundreds of millions of hours of training data, the model achieves, for the first time in the speech domain, low-sample generalization capabilities based on In-Context Learning (ICL), breaking the bottleneck of relying on large-scale labeled data in the speech field. Xiaomi-MiMo-Audio significantly outperforms open-source models with the same number of parameters in multiple standard benchmarks, achieving a best performance of 7B. It surpasses Google's Gemini-2.5-Flash on the standard test set of the audio understanding benchmark MMAU, and outperforms OpenAI's GPT-4o-Audio-Preview on the Big Bench Audio S2T task, a complex audio inference benchmark.

Xiaomi has open-sourced the pre-trained model MiMo-Audio-7B-Base and the instruction fine-tuning model MiMo-Audio-7B-Instruct, as well as a Tokenizer model with 1.2B parameters, supporting audio reconstruction and audio-to-text tasks.

Main functions of Xiaomi MiMo Audio

• Few-shot generalization abilityThis achievement marks the first time that In-Context Learning (ICL)-based few-shot generalization has been implemented in the speech domain, enabling rapid adaptation to new tasks and marking a "GPT-3 moment" in the speech field.
• Cross-modal alignment capabilityPost-training stimulates cross-modal alignment capabilities such as IQ, EQ, expressiveness, and safety, resulting in a high level of human-likeness in voice dialogue in terms of naturalness, emotional expression, and interactive adaptation.
• Speech understanding and generationIt significantly outperforms open-source models with the same number of parameters in multiple standard benchmarks such as general speech understanding and dialogue, achieving the best performance of 7B, and even surpassing some closed-source speech models.
• Audio Complex ReasoningIt performs exceptionally well on the Big Bench Audio S2T task, a benchmark for complex audio inference, demonstrating powerful capabilities in complex audio inference.
• Voice continuation abilityThe pre-trained model MiMo-Audio-7B-Base is currently the first open-source speech model with speech continuation capabilities.
• Support mixed thinkingIt is the first open-source model that simultaneously incorporates Thinking into both speech understanding and speech generation processes, supporting hybrid thinking.
• Audio to text taskThe Tokenizer model supports audio-to-text (A2T) tasks, covering over ten million hours of voice data.

The technical principle of Xiaomi-MiMo-Audio

• Innovative pre-training architectureIt employs an innovative pre-training architecture, trained on hundreds of millions of hours of training data, enabling the model to better handle speech data.
• Few-shot generalization abilityFor the first time, it has achieved a few-shot generalization capability based on In-Context Learning (ICL) in the speech domain, which can quickly adapt to new tasks with a small number of samples.
• Cross-modal alignment capabilityPost-training further enhances the model's cross-modal alignment capabilities, including intelligence, emotional intelligence, expressiveness, and safety, enabling speech dialogue to achieve a very high level of humanization in terms of naturalness, emotional expression, and interactive adaptation.
• Lossless compression pre-trainingBy using lossless speech compression pre-training, cross-task generalization was achieved, demonstrating the "emergent" behavior in the speech domain.
• Tokenizer modelIt employs a Transformer architecture Tokenizer model with 1.2B parameters, trained from scratch, covering over ten million hours of speech data, and supports audio reconstruction and audio-to-text (A2T) tasks.
• Lightweight post-trainingThe model performance is further optimized through lightweight post-training (SFT), making it excel in speech understanding and generation.
• Hybrid thinking mechanismBy introducing the Thinking mechanism into both speech understanding and speech generation processes, hybrid thinking is supported, which enhances the model's complex reasoning capabilities.

Xiaomi-MiMo-Audio project address

• Project official website: https://xiaomimimo.github.io/MiMo-Audio-Demo/
• Github repositoryhttps://github.com/XiaomiMiMo/MiMo-Audio
• HuggingFace model library
  • MiMo-Audio-7B-Basehttps://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Base
  • MiMo-Audio-7B-Instructhttps://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Instruct
  • Tokenizerhttps://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer
• Technical Papers: https://github.com/XiaomiMiMo/MiMo-Audio/blob/main/MiMo-Audio-Technical-Report.pdf

Application scenarios of Xiaomi-MiMo-Audio

• Voice interactionIt can be used in intelligent voice assistants to provide a more natural and intelligent voice interaction experience, and supports conversations in multiple languages and dialects.
• Speech generationIt can generate high-quality voice content, suitable for scenarios such as audiobooks, voice broadcasting, and voice navigation.
• Speech-to-textIt supports speech-to-text (A2T) tasks and can be applied to scenarios such as meeting minutes, voice input, and voice search.
• Audio content creationIt helps content creators generate audio scripts or voice content, improving their creation efficiency.
• Emotional expressionIt can express rich emotions in voice dialogue, making it suitable for scenarios that require emotional interaction, such as emotional companion robots and customer service systems.
• Speech Recognition and UnderstandingIt performs excellently in audio understanding benchmark tests and can be used in scenarios such as speech recognition and voice command control.