MiDashengLM - Xiaomi's open-source, high-efficiency large-scale sound understanding model

What is MiDashengLM?

MiDashengLM is a high-efficiency audio understanding model open-sourced by Xiaomi, specifically version MiDashengLM-7B. Built upon the Xiaomi Dasheng audio encoder and Qwen2.5-Omni-7B Thinker decoder, it employs a universal audio description alignment strategy to achieve a unified understanding of speech, ambient sounds, and music. The model boasts exceptional performance and high inference efficiency, with first-to-toe latency only one-quarter that of industry-leading models, supporting massively parallel processing. The training data is completely open-source, supporting academic and commercial applications, and is suitable for scenarios such as smart cockpits and smart homes, driving upgrades in multimodal interactive experiences.

Main functions of MiDashengLM

• Audio CaptioningIt converts audio content (including speech, ambient sound, music, etc.) into natural language descriptions to help users quickly understand audio information.
• Audio ClassificationIt identifies specific categories in audio (such as speech, ambient sound, music, etc.) and can be used in scenarios such as environmental sound recognition and music classification.
• Automatic Speech Recognition (ASR)It converts speech to text, supports multiple languages, and is widely used in scenarios such as voice assistants and smart cockpits.
• Audio Question AnsweringIt can answer related questions based on the input audio content, and is applicable to environmental sound Q&A, music Q&A, etc. in smart cockpits.
• Multimodal Interaction): Combining audio with other modalities (such as text and images) for comprehensive understanding, enhancing the interactive experience of smart devices.

MiDashengLM's technical principles

• Model Architecture:
  • Audio encoderBased on the Xiaomi Dasheng audio encoder, it is responsible for converting the input audio signal into a high-dimensional feature representation. The Dasheng encoder performs well in audio understanding tasks and can extract rich semantic information when processing non-speech audio (such as ambient sounds and music).
  • decoderBased on the Qwen2.5-Omni-7B Thinker autoregressive decoder, it is responsible for converting the features extracted by the audio encoder into natural language descriptions. The decoder supports various tasks, including audio description, audio question answering, and speech recognition.
• Training strategy:
  • General audio description alignmentBased on a general audio description alignment strategy, this approach avoids the limitations of traditional ASR transcription methods. Through non-monotonic global semantic mapping, it forces the model to learn deep semantic associations in audio scenes, achieving a unified understanding of speech, environmental sounds, and music.
  • Multiple expert analysisThe training data is generated based on a multi-expert analysis pipeline, including fine-grained annotations of speech, human voice, music, and ambient acoustics, and then synthesized into a unified description using a large DeepSeek-R1 inference model.
  • DatasetThe dataset was trained using publicly available datasets covering multiple domains, including speech, ambient sound, and music, totaling over 1 million hours. The original labels from the training data were discarded during pre-training, and only new, richer textual descriptions were used, forcing the model to learn richer and more comprehensive sound information.
• Inference efficiency optimization:
  • Efficient ReasoningBy optimizing the audio encoder design, the output frame rate was reduced from 25Hz in Qwen2.5-Omni to 5Hz, significantly reducing computational load and improving inference efficiency.
  • Massive parallel processingIt supports larger batch processing (batch size=512). When processing 30 seconds of audio and generating 100 tokens on an 80GB GPU, the throughput of MiDashengLM is more than 20 times that of Qwen2.5-Omni-7B.

MiDashengLM's project address

• GitHub repositoryhttps://github.com/xiaomi-research/dasheng-lm
• HuggingFace model libraryhttps://huggingface.co/mispeech/midashenglm-7b
• Technical Papers: https://github.com/xiaomi-research/dasheng-lm/blob/main/technical_report/MiDashengLM_techreport.pdf
• Experience the demo online:https://huggingface.co/spaces/mispeech/MiDashengLM-7B

Application Scenarios of MiDashengLM

• Smart CockpitBased on voice assistant and environmental sound recognition, it enhances driving safety and interactive experience.
• Smart HomeEnables convenient home automation through voice control and environmental sound monitoring.
• voice assistantIt provides multilingual speech recognition and intelligent dialogue to meet diverse user needs.
• Audio content creation and annotationAutomatically generates audio descriptions and tags, improving content creation efficiency.
• Education and LearningIt assists users in language and music learning by providing pronunciation feedback and theoretical guidance.