MinMo - A large-scale multimodal voice interaction model launched by Alibaba Tongyi Labs

What is MinMo?

MinMo, developed by the FunAudioLLM team at Alibaba's Tongyi Lab, is a multimodal large-scale model focused on achieving seamless voice interaction. MinMo boasts approximately 8 billion parameters, trained through multi-stage training on 1.4 million hours of diverse speech data and a wide range of speech tasks. MinMo supports user-instructed control over the emotion, dialect, and speaking style of generated audio, as well as mimicking specific timbres, achieving a generation efficiency exceeding 90%. MinMo supports full-duplex voice interaction, with a speech-to-text latency of approximately 100 milliseconds. Theoretically, the full-duplex latency is approximately 600 milliseconds, but in practice, it is approximately 800 milliseconds, enabling simultaneous bidirectional communication between the user and the system, making multi-turn dialogues smoother.

MinMo's main functions

• Real-time voice dialogueIt can engage in real-time, natural, and fluent voice conversations with users, understand their voice commands, and generate corresponding voice responses.
• Multilingual supportIt supports multilingual speech recognition and translation, enabling smooth communication with users in various language environments.
• Emotional expressionGenerates voice with specific emotions (such as happiness, sadness, surprise, etc.) based on user instructions.
• Dialect and speaking styleIt supports generating speech in specific dialects (such as Sichuanese, Cantonese, etc.) and specific speaking styles (such as fast, slow, etc.).
• Timbre ImitationIt can mimic specific timbres to make voice interaction more personalized and expressive.
• Full-duplex interactionIt supports simultaneous speaking and listening by both the user and the system, enabling more natural and efficient multi-turn dialogues. The speech-to-text latency is approximately 100 milliseconds, and the full-duplex latency is theoretically approximately 600 milliseconds, but in practice, it is approximately 800 milliseconds.

MinMo's technical principles

• Multimodal fusion architecture:
  • Voice encoderBased on the pre-trained SenseVoice-large encoder module, it provides powerful speech understanding capabilities and supports multilingual speech recognition, emotion recognition, and audio event detection.
  • Input ProjectorIt consists of two Transformer layers and one CNN layer, and is used for dimension alignment and downsampling.
  • Large language modelThe pre-trained Qwen2.5-7B-instruct model was used because of its excellent performance on multiple benchmark tests.
  • Output projector: Single-layer linear module, used for dimension alignment.
  • Speech markup language modelSpeech tags are generated autoregressively using a pre-trained CosyVoice 2 LM module.
  • Token2wav synthesizerIt converts speech tokens into MEL spectrograms, then into waveforms, supporting real-time audio synthesis.
  • Full-duplex predictorA single-layer Transformer and linear softmax output layer used to predict in real time whether to continue system response or pause processing of user input.
• Multi-stage training strategy:
  • Speech-to-text alignmentBased on a large amount of speech data and corresponding text annotations, the model is trained to learn the mapping relationship between speech and text, enabling the model to accurately convert speech into text, laying the foundation for subsequent text understanding and generation.
  • Text-to-speech alignmentThe goal is to enable the model to learn how to convert text into speech, generating natural and fluent speech expressions while preserving the semantic information and emotional nuances of the text.
  • Voice-to-voice alignmentThis further enhances the model's ability to understand and generate speech, allowing the model to interact directly at the speech level and better handle features such as prosody and intonation.
  • Duplex Interaction AlignmentSimulate real full-duplex interaction scenarios, train the model to accurately perform speech recognition and generation when simultaneously receiving and sending speech signals, and optimize the model's performance in complex interaction environments.

MinMo's project address

• Project official website:https://funaudiollm.github.io/minmo/
• arXiv technical paper:https://arxiv.org/pdf/2501.06282

MinMo Application Scenarios

• Intelligent Customer ServiceIt offers 24/7 multilingual voice support, real-time interactive answers to customer questions, personalized services based on emotion recognition, and supports full-duplex dialogue to improve efficiency.
• Smart AssistantControl smart home devices, manage schedules, query information, recommend personalized content, and improve the convenience of life and the efficiency of information access.
• EducationIt assists in language learning, enhances engagement through interactive teaching, provides personalized plans based on learning progress, and offers emotional support to encourage student learning.
• HealthcareRemote medical consultation, health monitoring reminders, rehabilitation training guidance, and emotional support and counseling enhance the accessibility of medical services and the patient experience.
• Intelligent drivingThe voice-controlled vehicle system provides real-time traffic information, emergency guidance, and full-duplex dialogue to improve driving safety and convenience.