MiniCPM-o 4.5 - Wallfacer Intelligence's Open Source Full-Duplex Full-Modal Model

What is MiniCPM-o 4.5?

MiniCPM-o 4.5 is Mianbi Intelligent's open-source 9B-parameter full-modal flagship model, employing an end-to-end architecture that integrates SigLip2, Whisper, CosyVoice2, and Qwen3-8B. As the industry's first model to support "real-time free dialogue," it achieves full-duplex interaction—allowing users to see, hear, and speak simultaneously, moving away from the traditional turn-based "walkie-talkie" mode. The model boasts leading visual understanding, hyper-humanoid speech generation, and voice cloning capabilities, supporting proactive interaction and real-time streaming media processing. It can run on edge devices and is compatible with various domestic chips such as Ascend and Hygon, achieving efficient deployment through frameworks like llama.cpp and vLLM.

Main functions of MiniCPM-o 4.5

• Full-duplex real-time interactionThe model can simultaneously process visual and audio inputs and generate speech outputs, enabling parallel perception and expression that allows users to see, hear, and speak at the same time.
• Proactive intelligent interactionThe model autonomously monitors environmental changes once per second, proactively determines when to speak, and enables human-like interactive behaviors such as proactive reminders and real-time comments.
• Hyperhuman-like speech synthesisIt supports end-to-end speech generation with rich emotions and natural timbre, and can clone customized voices based on a few seconds of audio samples, while maintaining stability and consistency in long speech synthesis.
• Leading visual understandingIn the OpenCompass benchmark, it outperformed GPT-4o and Gemini 2.0 Pro with 9B parameters, supporting high-resolution image analysis and real-time understanding of high-frame-rate video.
• End-to-end document parsingIt achieves industry-leading performance on the OmniDocBench benchmark, efficiently handling tasks such as understanding and extracting structured English documents with complex layouts.

Technical Principles of MiniCPM-o 4.5

• End-to-end full-modal architecture designMiniCPM-o 4.5 uses a dense feature connection to perform end-to-end joint training of the SigLip2 visual encoder, Whisper-medium audio encoder, CosyVoice2 speech decoder and Qwen3-8B language model. The tightly coupled design allows information from each modality to flow freely within the model, avoiding information loss and error accumulation in traditional pipeline architectures, and achieving more accurate multimodal understanding and generation control.
• Full-duplex multimodal real-time streaming mechanismThe model transforms the offline modal codec into an online version that supports streaming input and output. The speech decoder uses a text and speech token interleaving modeling approach to achieve full-duplex generation. During inference, a time-division multiplexing mechanism divides the parallel multimodal data stream into sequential information groups within millisecond-level time slices, enabling the language model backbone to be uniformly scheduled and processed, efficiently completing the synchronous perception and response of real-time audio and video streams within a single architecture.
• Proactive interactive decision-making mechanismThe language model module continuously monitors the input video and audio streams and automatically triggers speech decisions at a frequency of 1Hz. The high-frequency decision-making capability combined with full-duplex characteristics enables the model to autonomously select the most appropriate time and content to respond based on dynamic changes in the environment, breaking through the limitations of traditional models that passively wait for user instructions.
• Configurable speech modeling designThe model continues the design paradigm of multimodal system cues, supporting both text and audio system cues as input. Audio system cues are used to specify target timbre features. This design allows the model to perform voice cloning and role-playing by providing only a short reference audio sample during the inference phase.

MiniCPM-o 4.5 project address

• GitHub repositoryhttps://github.com/OpenBMB/MiniCPM-o
• HuggingFace model libraryhttps://huggingface.co/openbmb/MiniCPM-o-4_5
• Experience the demo onlinehttps://huggingface.co/spaces/openbmb/minicpm-omni

Application scenarios of MiniCPM-o 4.5

• Smart assistants and companionsAs an all-around AI assistant, the model can perceive the user's environment and emotions in real time, proactively provide reminders, suggestions or emotional companionship, support personalized voice cloning, and create a unique interactive experience.
• Real-time video interactionIt is suitable for scenarios such as video surveillance analysis, live broadcast commentary, and remote teaching and tutoring. It can simultaneously understand the content on the screen and voice commands, and provide real-time voice feedback.
• Intelligent customer service and shopping guideIt provides natural and fluent voice services in e-commerce, finance, government affairs and other fields, supports multi-turn dialogue and proactive recommendations, and improves user service experience and business conversion efficiency.
• Education and TrainingIt is used for language learning practice, virtual teachers, skills training, etc., and achieves immersive interactive teaching through the combination of visual demonstrations and voice explanations.
• Content creation and entertainmentIt supports audiobook generation, virtual character voice acting, game NPC interaction, etc. The voice cloning function can quickly replicate the voice of a specific character for role-playing.