WorldPM - A series of preference modeling models jointly launched by Alibaba's Qwen team and Fudan University.
WorldPM (World Preference Modeling) is a series of preference modeling models developed by Alibaba Group's Qwen team and Fudan University. It reveals the scalability of preference models through large-scale training. The model is based on 15 million...
What is WorldPM?
WorldPM (World Preference Modeling) is a series of preference modeling models developed by Alibaba Group's Qwen team and Fudan University. It reveals the scalability of preference models through large-scale training. The model was trained on 15 million preference data points, finding that preference models exhibit a clear power-law decreasing trend in the objective domain, while in the subjective domain, due to their multi-dimensional nature, they struggle to show a single scalability trend. The project provides a base model of 72B size and several versions fine-tuned for specific datasets. WorldPM-72B-HelpSteer2 is suitable for specific tasks requiring precise preference judgments, WorldPM-72B-UltraFeedback is suitable for handling large-scale user feedback scenarios, and WorldPM-72B-RLHFLow is designed for low-resource preference modeling, suitable for scenarios with limited data. It supports rapid deployment based on Hugging Faces. WorldPM provides powerful preference modeling capabilities for tasks such as dialogue systems and recommendation systems in natural language processing, contributing to the development of related fields.
Main functions of World PM
- Preference modelingLearn human preference patterns to form a unified preference representation.
- Improve generalization ability: Helps the model perform better on different datasets.
- As a basic modelUsed for preference fine-tuning to improve model performance.
- Enhance robustness: Identify and process errors or incomplete information.
The technical principles of WorldPM
- Preference data collection and processingWorldPM collects user-generated preference data from public forums such as StackExchange and Reddit. The data naturally forms preference pairs based on user voting mechanisms for different answers.
- Massive training and the law of scaleWorldPM borrows the scaling law from language modeling, where model performance improves with increasing model parameter size and training data volume. By training on models of varying sizes (from 1.5B to 72B parameters), WorldPM explores the scaling potential of preference modeling. Experiments show that in both adversarial and objectivity assessment tasks, model performance significantly improves with increasing training data and model size.
- Preference Modeling FrameworkWorldPM is a modeling framework based on binary preference pairs. For each pair of preference samples, the model calculates a reward score for each answer and optimizes the BT loss function based on the Bradley-Terry model to learn preference patterns.
- Style Preference Analysis and ControlTo address the style preference issue in subjective preference assessment, a style preference analysis and control mechanism is introduced. By separating style features (such as text length, Markdown format, etc.) from content features, the model can more accurately assess preferences and reduce the interference of style factors on the assessment results.
- Model expansion and fine-tuningWorldPM can be used directly for preference evaluation or as the initialization basis for other models for further preference fine-tuning. By fine-tuning on human preference datasets of varying sizes, WorldPM can further improve model performance, especially showing significant performance improvements when data is limited.
WorldPM's project address
- GitHub repository:https://github.com/QwenLM/WorldPM
- HuggingFace model library:https://huggingface.co/Qwen/WorldPM-72B
- arXiv technical paper:https://arxiv.org/pdf/2505.10527
Application scenarios of WorldPM
- Language generation optimizationTo make AI-generated text more natural and in line with human preferences, such as improving the quality of chatbot responses.
- Personalized recommendationsRecommend content based on user preferences, such as articles, videos, or music, to improve the accuracy and satisfaction of recommendations.
- Improvements to intelligent customer serviceIt helps intelligent customer service better understand user needs, generate responses that better meet user expectations, and improve user experience.
- Content moderation and securityIt identifies and filters out erroneous or harmful information to ensure the security and reliability of content.
- Multimodal applicationsExtend to preference modeling of multimodal content such as images and videos to optimize content generation and moderation.