R1-Omni - A multimodal large language model open sourced by Alibaba Tongyi

What is R1-Omni?

R1-Omni, developed by Alibaba Tongyi, is a multimodal language model based on reinforcement learning (RLVR) and focuses on emotion recognition tasks. By combining visual and audio information, it can clearly explain the reasoning process of emotion recognition, demonstrating powerful emotion understanding capabilities. On multiple emotion recognition datasets, R1-Omni significantly outperforms supervised fine-tuning (SFT) models, performs well in out-of-distribution scenarios, and has strong generalization ability.

Main functions of R1-Omni

• Multimodal sentiment analysisThe R1-Omni can process visual and audio information simultaneously, performing emotion recognition on input video or audio content and accurately judging the emotions expressed within it.
• Explainable reasoning processThe model provides emotion recognition results and generates a detailed reasoning process. This reasoning process is encapsulated within specific labels, explaining how the model integrates visual and audio cues to arrive at predictions, thus enhancing the model's interpretability.
• Training based on RLVRR1-Omni employs the RLVR training paradigm, directly evaluating the output based on a verifiable reward function without relying on a separate reward model. This simplifies the reward mechanism while ensuring consistency with the task's inherent correctness standards.
• Application of GRPO methodBy combining the Generative Relative Policy Optimization (GRPO) method, the generated response groups are directly compared, avoiding the use of an additional critic model and enhancing the model's ability to distinguish between high-quality and low-quality outputs.
• Enhanced reasoning abilityCompared to other baseline models, R1-Omni provides a more coherent, accurate, and interpretable reasoning process.
• Improved comprehensionOn multiple emotion recognition datasets, R1-Omni significantly outperforms other models in emotion recognition accuracy.
• Stronger generalization abilityOn out-of-distribution (OOD) datasets, R1-Omni performs well and is better adapted to unseen scenarios.

R1-Omni's technical principles

• RLVR Training ParadigmRLVR is a novel training paradigm. Its core idea is to directly evaluate the model output based on a validation function, eliminating the need for a separate reward model in traditional human feedback reinforcement learning (RLHF). Given an input problem q, the policy model πθ generates a response o, which is then evaluated using a verifiable reward function R(q,o). The optimization objective is to maximize the validation reward minus a regularization term based on KL divergence. This simplifies the reward mechanism while ensuring consistency with the inherent correctness criteria of the task.
• GRPO methodGRPO (Grouped Relative Policy Optimization) is an improved reinforcement learning method, unlike the traditional PPO (Proximal Policy Optimization). GRPO avoids using an additional critic model and directly compares the generated response groups. Specifically, for a problem q, multiple output groups {o1​,o2​,…,oG​} are sampled, a reward score {r1​,r2​,…,rG​} is calculated for each output, and these reward scores are then normalized to form a relative reward. This more directly reflects the relative merits of different outputs for the same problem, enhancing the model's ability to distinguish between high-quality and low-quality outputs.
• Cold start strategyThe R1-Omni model was built using a cold-start strategy inspired by DeepSeek-R1. First, HumanOmni-0.5B was fine-tuned on a combined dataset containing 232 samples from the Explainable Multimodal Sentiment Inference (EMER) dataset and 348 manually labeled samples from the HumanOmni dataset. This gave the model preliminary reasoning capabilities, allowing it to understand how visual and audio cues influence sentiment recognition. The model was then further optimized through RLVR training.
• Reward function designDuring RLVR training, the reward function consists of two parts: accuracy reward and format reward. The accuracy reward evaluates the match between the predicted sentiment and the real sentiment, while the format reward ensures that the model output conforms to the specified HTML tag format. This guarantees the accuracy of sentiment recognition and the interpretability of the output.
• Model output formatThe output of R1-Omni is expected to consist of two parts: the reasoning process and the final sentiment label. The reasoning process is encapsulated within the `<think></think>` tag, explaining how the model integrates visual and audio cues to arrive at a prediction; the final sentiment label is encapsulated within the `<answer></answer>` tag, representing the predicted sentiment. Providing sentiment recognition results and a detailed reasoning process enhances the model's interpretability.

R1-Omni project address

• Github repository:https://github.com/HumanMLLM/R1-Omni
• HuggingFace model library:https://huggingface.co/StarJiaxing/R1-Omni-0.5B
• arXiv technical paper:https://arxiv.org/pdf/2503.05379

Application scenarios of R1-Omni

• Sentiment AnalysisIt can be used in scenarios such as social media management, public opinion monitoring, and consumer sentiment analysis to help businesses interact with target users more effectively.
• Content creation assistanceIt combines AI drawing and writing tools to provide high-quality solutions for marketing, advertising creativity, and more.
• Mental health assessmentR1-Omni can analyze patients' emotional expressions, assisting mental health professionals in assessment and intervention.
• EducationIn online education, R1-Omni can analyze students' emotional responses and help teachers adjust their teaching strategies.