Skywork-Reward-V2 - Kunlun Wanwei's open-source second-generation reward model series

What is Skywork-Reward-V2?

Skywork-Reward-V2 is Kunlun Wanwei's second-generation open-source reward model series, comprising eight models based on different pedestal models and sizes, with parameter scales ranging from 600 million to 8 billion. The Skywork-Reward-V2 series models have topped all seven major reward model benchmarks, demonstrating outstanding performance. The model's success is attributed to the Skywork-SynPref-40M dataset, a mixed dataset containing 40 million pairs of preference samples, meticulously selected and filtered through a two-stage human-machine collaborative process. Skywork-Reward-V2 excels in general preference alignment, objectivity, and security, demonstrating strong generalization capabilities in Best-of-N scalability and style bias resistance.

Main functions of Skywork-Reward-V2

• General Preference AlignmentAccurately determine which of the different answers better reflects general human preferences, allowing the model's output to more closely align with human subjective judgment, such as selecting a more natural and appropriate response in a chat scenario.
• Objective accuracy assessmentIt effectively identifies the objectivity and accuracy of answers. For questions with clear factual basis, it can filter out the correct answers, such as judging the accuracy of answers in tasks like mathematical calculations and fact-finding.
• Security assessmentIt has the ability to identify whether the answer is safe, avoid generating responses containing harmful or inappropriate content, and ensure that the model output meets ethical and safety standards, such as filtering out answers involving violence, discrimination, or other inappropriate content.
• Best-of-N scalabilityWhen faced with multiple candidate answers, it can efficiently select the optimal answer, improving the model's decision-making ability in multi-choice scenarios, such as providing the best solution for users in multi-turn dialogues.
• Style Deviation ResistanceIt exhibits strong adaptability and fairness to responses of different styles, and will not be biased due to differences in response style. It ensures that the model maintains objective evaluation in diverse expressions, such as making reasonable judgments in different styles of texts such as literary creations and professional discourses.

The technical principles of Skywork-Reward-V2

• Skywork-SynPref-40M, a large-scale, high-quality datasetIt contains 40 million pairs of preference samples, providing a rich data foundation for model training. Based on a two-stage human-machine collaborative process, combining the high quality of manual annotation with the scalable processing capabilities of the model, 26 million high-quality preference data points were selected from a large amount of data, ensuring the diversity and accuracy of the data.
• Training based on the Bradley-Terry modelThe reward model is trained by calculating relative preference scores between different answers, based on the classic Bradley-Terry model. During training, the model continuously learns how to rank answers according to human preferences, optimizes the reward signal, and allows the model to better capture human preference characteristics.
• Multi-round iterative training and optimizationBased on multi-round iterative training, the model identifies weaknesses in each round based on its current performance, and further expands and enhances the training data by retrieving similar samples and automatically labeling them using a multi-model consensus mechanism. The iterative optimization process continuously improves the model's ability to understand and discriminate preferences, resulting in excellent performance on different benchmarks.
• Model architecture and parameter tuningBased on the Qwen3 and LLaMA3 series models, this system provides models with different parameter scales to meet the needs of various scenarios. By adjusting model parameters and training strategies, such as learning rate and batch size, model performance is optimized, ensuring efficient convergence and stable performance under large-scale data training.

Skywork-Reward-V2 project address

• GitHub repository: https://github.com/SkyworkAI/Skywork-Reward-V2
• HuggingFace model library: https://huggingface.co/collections/Skywork/skywork-reward-v2-685cc86ce5d9c9e4be500c84
• arXiv technical paperhttps://arxiv.org/pdf/2507.01352

Application scenarios of Skywork-Reward-V2

• Dialogue system optimizationEvaluate the responses generated by the dialogue system, select answers that better align with human language habits and preferences, and improve the interaction quality and user experience of systems such as intelligent customer service and chatbots.
• Content recommendation optimizationIn content recommendation systems, the attractiveness and relevance of different content items are evaluated, and more accurate and personalized recommendations are provided to users based on their historical behavior and preferences, thereby improving the accuracy of the recommendation system and user satisfaction.
• Educational guidance and supportIn the field of education, it involves evaluating students' submitted answers, judging their accuracy and completeness, providing targeted feedback and guidance, and assisting teachers in conducting teaching assessments.
• Content moderationIt detects and filters harmful, inappropriate, or illegal content, such as violence, pornography, and discrimination, to ensure the safe and compliant operation of social media, forums, and other platforms.
• Game optimizationIn game development, we evaluate and optimize textual content such as game plot, character dialogue, and quest design to enhance the game's immersion and enjoyment.