Verifier Engineering - A new post-training paradigm jointly launched by the Chinese Academy of Sciences, Alibaba, and Xiaohongshu.
Verifier Engineering is a novel post-training paradigm jointly launched by the Chinese Academy of Sciences, Alibaba, and Xiaohongshu. It addresses the challenge of providing effective supervisory signals in basic model design. Verifier Engineering...
What is Verifier Engineering?
Verifier Engineering, a novel post-training paradigm jointly launched by the Chinese Academy of Sciences, Alibaba, and Xiaohongshu, addresses the challenge of providing effective supervisory signals in basic model design. Based on automated verifiers, Verifier Engineering performs verification tasks and provides feedback to the base model, consisting of three phases: search, verification, and feedback, to optimize model performance. Based on a closed-loop feedback mechanism, it enhances the model's generalization ability, representing a crucial step in achieving artificial general intelligence.
The main functions of Verifier Engineering
- SearchBased on given instructions, sample representative or potential problem samples from the model output distribution.
- VerifyUse various validators (such as rule detection, evaluation metrics, or manual annotation) to evaluate the generated candidate responses.
- FeedbackEnhance model performance using methods such as supervised fine-tuning based on validation results or contextual learning.
- Improve model performanceBased on a closed-loop feedback loop, the accuracy and reliability of the model are continuously improved.
Verifier Engineering's Technical Principles
- Objective-Condition Markov Decision Process (GC-MDP)The validator project is formalized as a GC-MDP, which includes state space, action space, transition function, target space, target distribution, and reward function.
- Search structureLinear search and tree search, navigating in state-action space, balancing exploration and exploitation.
- Validator ClassificationThe criteria for validation are: binary, score, ranking, text feedback, validation granularity (label level, thought level, trajectory level), validator source (program-based, model-based), and whether additional training of a classification validator is required.
- Feedback methods:
- Training-based feedback: Efficiently update model parameters based on data.
- Feedback based on inference: Modifying the input or strategy during inference without changing the model parameters.
Verifier Engineering's project address
- GitHub repository:https://github.com/icip-cas/Verifier-Engineering
- HuggingFace model library:https://huggingface.co/papers/2411.11504
- arXiv technical paper:https://arxiv.org/pdf/2411.11504
Application scenarios of Verifier Engineering
- Natural Language Processing (NLP)Machine translation, text summarization, sentiment analysis, question answering systems, dialogue systems, etc., enhance the language understanding and generation capabilities of models based on validator engineering.
- Code generation and programming assistanceIt assists developers in writing and debugging code, automatically generates code snippets, and optimizes code quality.
- Education and academic researchBuild an intelligent education assistant that provides personalized learning suggestions and automatically evaluates student assignments and papers.
- Content moderation and securityDetect and filter inappropriate content, such as hate speech, misinformation, and content that violates policies, to ensure the safety of platform content.
- Q&A and search enginesTo improve the accuracy of search engines and provide more relevant and accurate information retrieval results.