Skywork-VL Reward - Skywork AI's open-source multimodal reward model
Skywork-VL Reward is an open-source multimodal reward model from Skywork AI, providing reliable reward signals for multimodal understanding and reasoning tasks. The model is based on the Qwen2.5-VL-7B-Instruct architecture, and incorporates a reward head structure...
What is Skywork-VL Reward?
Skywork-VL Reward is an open-source multimodal reward model from Skywork AI, providing reliable reward signals for multimodal understanding and reasoning tasks. Based on the Qwen2.5-VL-7B-Instruct architecture, the model incorporates a reward head structure, is trained using pairwise preference data, and outputs a scalar reward score aligned with human preferences. The model achieves a state-of-the-art (SOTA) score of 73.1 on VL-RewardBench and performs exceptionally well on RewardBench, reaching a high score of 90.1. Skywork-VL Reward significantly enhances multimodal reasoning capabilities based on Hybrid Preference Optimization (MPO), representing a new breakthrough in the field of multimodal reinforcement learning.
Main functions of Skywork-VL Reward
- Evaluate multimodal output: Evaluate the quality of the output generated by the Visual-Language Model (VLM) to determine whether it conforms to human preferences.
- Provide reward signalsOutputs a scalar reward score, reflecting the quality of the generated content or its alignment with human preferences.
- Support multimodal tasksIt is applicable to a variety of multimodal tasks, such as image description and complex reasoning, and has wide applicability.
- Improve model performanceBased on generating high-quality preference data, it supports hybrid preference optimization (MPO), significantly improving multimodal reasoning capabilities.
The technical principles of Skywork-VL Reward
- Model ArchitectureBased on the Qwen2.5-VL-7B-Instruct architecture, the architecture includes a visual encoder (Vision Transformer), a vision-language adapter, and a language model decoder. A reward head structure is added to the base model to output a scalar reward score. The reward head processes the final hidden state based on fully connected layers to generate the reward score.
- Dataset ConstructionThis approach integrates multiple open-source preference datasets (such as LLaVA-Critic-113k, Skywork-Reward-Preference-80K-v0.2, and RLAIF-V-Dataset) and internally labeled complex inference task data. Through steps such as deduplication, similarity filtering, and preference judgment filtering, it ensures high data quality and consistency. A high-quality preference data is generated using an advanced VLM inference engine, enhancing the model's generalization ability.
- Training methodsBased on the pairwise preference loss function, the model is trained by comparing the merits of two candidate responses, allowing the model to learn relative rankings. Two-stage fine-tuning is employed: the first stage uses multimodal preference data for training, and the second stage incorporates plain text preference data to further improve the model's performance in plain text scenarios.
Skywork-VL Reward project address
- HuggingFace model library:https://huggingface.co/Skywork/Skywork-VL-Reward
- arXiv technical paper:https://arxiv.org/pdf/2505.07263
Application scenarios of Skywork-VL Reward
- Content generation assessmentTo evaluate the quality of multimodal content generation, such as image descriptions and video captions, and to determine whether the generated content is accurate and conforms to human preferences.
- Reasoning task optimizationIn complex multimodal reasoning tasks, such as visual question answering and geometric problems, it evaluates the rationality of the reasoning process and results, and helps to optimize the reasoning model.
- Model AlignmentEnsure that the output of the multimodal model is aligned with human values and ethical standards to avoid generating harmful or misleading content.
- Mixed Preference Optimization (MPO)As a key component of MPO training, it provides high-quality preference data, improving the inference ability and generalization performance of multimodal models.
- BenchmarkingAs a benchmark tool for multimodal tasks, it evaluates and compares the performance of different models, and promotes the development of multimodal technology.