HumanOmniV2 - A multimodal inference model open sourced by Alibaba Tongyi

What is HumanOmniV2?

HumanOmniV2 is an open-source multimodal reasoning model from Alibaba's Tongyi Lab. Based on a forced context summarization mechanism, a large-model-driven multi-dimensional reward system, and a GRPO-based optimization training method, the model addresses the issues of insufficient global context understanding and simplistic reasoning paths in multimodal reasoning. Before generating answers, the model systematically analyzes visual, auditory, and linguistic signals to construct a complete scene background, accurately capturing hidden logic and deep intentions within multimodal information. HumanOmniV2 performs exceptionally well in benchmark tests such as IntentBench, achieving an accuracy of 69.33%, providing important reference for AI to understand complex human intentions. The model is now open-source for research and application.

Main functions of HumanOmniV2

• Comprehensive understanding of multimodal informationIt comprehensively analyzes visual, auditory, and linguistic signals from various input formats such as images, videos, and audio to capture hidden information and deep logic.
• Accurate inference of human intentionsBased on a systematic analysis of the context, accurately understand the true intentions in a dialogue or scenario, including complex emotions, social relationships, and potential biases.
• Generate structured reasoning pathsDuring the reasoning process, the model outputs a detailed summary of the context and reasoning steps, ensuring that the reasoning process is transparent and interpretable.
• Coping with complex social scenariosIn complex social interactions, it identifies and understands people's emotions, behavioral motivations, and social relationships, providing judgments that are more consistent with human cognition.

The technical principles of HumanOmniV2

• Forced context summarization mechanismBefore generating the final answer, the model outputs a context summary within a `<context>` tag to ensure that key information in the multimodal input is not skipped. This structured design helps the model systematically analyze visual, auditory, and linguistic signals to construct a complete scene context.
• A multi-dimensional reward system driven by a large modelContextual rewards assess the model's accuracy in understanding the overall context of multimodal input. Format rewards ensure the model's output conforms to structured requirements. Accuracy rewards improve the model's correctness of responses. Logical rewards incentivize the model to use advanced reasoning methods such as reflection, induction, and deduction, avoiding simple reliance on textual reasoning.
• GRPO-based optimization training method:
  • Introducing word-level loss(Token-level Loss): Solves the imbalance problem in long sequence training.
  • Remove problem-level normalization termsTo avoid weight bias between samples of different difficulty.
  • Applying dynamic KL divergence mechanismEncourage exploration in the early stages of training and achieve stable convergence in the later stages to improve the model's generalization ability and training stability.
• High-quality full-modal inference training datasetWe construct high-quality datasets containing image, video, and audio tasks, along with detailed context summaries and inference path annotations, providing a solid foundation for cold-start training and reinforcement learning of the model.
• The new benchmark IntentBenchIt includes 633 videos and 2,689 related questions, closely linked to auditory and visual cues in the videos, and focuses on assessing the model's ability to deeply understand human behavioral motivations, emotional states, and social interactions.

HumanOmniV2 project address

• GitHub repository: https://github.com/HumanMLLM/HumanOmniV2
• HuggingFace model libraryhttps://huggingface.co/PhilipC/HumanOmniV2
• arXiv technical paper: https://arxiv.org/pdf/2506.21277

Application scenarios of HumanOmniV2

• Video content understanding and recommendationAnalyze the emotions, relationships between characters, and background scenes in videos to provide accurate content recommendations for video platforms, helping users discover videos that better match their interests and emotions.
• Intelligent customer service and customer experience optimizationBy analyzing customer emotions and needs through voice and text, it provides real-time feedback to the customer service system, helping customer service personnel to better handle customer issues and improve customer satisfaction.
• Emotion Recognition and Mental Health SupportBy combining voice tone, facial expressions, and language content, it can identify the user's emotional state and assist mental health applications in providing more accurate emotional support and intervention suggestions.
• Social interaction analysis and optimizationAnalyze interactive content on social media platforms to identify potential misunderstandings or conflicts, help optimize social recommendations and user interaction experience, and improve the harmony of social media platforms.
• Education and Personalized LearningThe goal is to analyze students' emotions and behaviors during the learning process, provide personalized learning suggestions for online education platforms, and help teachers optimize teaching content and methods to improve learning outcomes.