OThink-MR1 - A multimodal language model optimization framework jointly developed by OPPO and HKUST.
OThink-MR1 is a multimodal language model optimization framework jointly developed by OPPO Research Institute and Hong Kong University of Science and Technology (Guangzhou). Based on dynamically adjusting the Kullback-Leibler (KL) divergence policy (GRPO-D) and reward model, it improves...
What is OThink-MR1?
OThink-MR1 is a multimodal language model optimization framework jointly developed by OPPO Research Institute and Hong Kong University of Science and Technology (Guangzhou). Based on dynamically adjusting the Kullback-Leibler (KL) divergence policy (GRPO-D) and reward model, it enhances the generalization reasoning ability of multimodal models in complex tasks. OThink-MR1 performs exceptionally well in multimodal tasks such as visual counting and geometric reasoning, outperforming traditional supervised fine-tuning (SFT) methods in same-task validation and demonstrating strong adaptability in cross-task generalization experiments. OThink-MR1 opens a new path for the development of general reasoning capabilities in multimodal models and is expected to play an important role in more fields.
Main functions of OThink-MR1
- Improve multimodal task performanceBased on dynamic reinforcement learning optimization model, it significantly improves the accuracy and generalization ability of multimodal tasks (such as visual counting, geometric reasoning, etc.).
- Cross-task generalization capabilityThe goal is to enable models trained on one type of multimodal task to be effectively transferred to other different types of multimodal tasks, thereby reducing dependence on data specific to a particular task.
- Exploration and utilization of dynamic equilibriumDuring training, the balance between exploring new strategies and using existing experience is dynamically adjusted to improve the model's global optimization capabilities.
- Enhance the model's reasoning abilityBased on the reward model, the model is guided to generate accurate and formatted output, thereby improving the overall reasoning ability.
OThink-MR1 Technical Principles
- Dynamic KL divergence strategy (GRPO-D)The GRPO-D strategy is inspired by the ϵ-greedy strategy in classic reinforcement learning, following the principle of "early exploration, later utilization." It balances the model's exploration (trying new strategies) and utilization (using existing experience) during training by dynamically adjusting the weights of the KL divergence. In the early stages of training, the KL divergence weights are small, encouraging the model to explore extensively; as training progresses, the weights gradually increase, guiding the model to utilize accumulated experience and avoiding premature convergence to a suboptimal solution.
- Reward ModelEvaluate the accuracy of the model's output, such as how well the model's output matches the actual count in a visual counting task. Ensure the model's output conforms to specific format requirements, such as whether the format of the model's output is correct in a geometric reasoning task. Combine validation accuracy rewards and format rewards to provide the model with more comprehensive feedback and guide its learning process.
- Reinforcement learning optimizationBased on maximizing the reward function, the model's strategy is optimized. In each training step, the model generates output according to the current strategy, the reward model evaluates the quality of the output, and the model adjusts its strategy according to the reward signal to gradually improve performance.
OThink-MR1 project address
- arXiv technical paper:https://arxiv.org/pdf/2503.16081
Application scenarios of OThink-MR1
- Intelligent visual question answeringAccurately understand image content and generate answers, such as identifying the number of objects in complex scenes.
- Image description generationGenerates rich and accurate image descriptions, providing more detailed visual information.
- Solving geometric problemsAnalyze the geometric shapes in an image and calculate geometric properties such as angles and lengths.
- Multimodal content moderationBy combining image and text information, we can determine whether the content complies with regulations and improve review efficiency.
- Virtual Reality and Augmented RealityProvides users with an intelligent interactive experience, such as real-time scene interpretation and navigation suggestions.