Self-Taught Evaluators - A New Model Evaluation Method Introduced by Meta
Self-Taught Evaluators are a novel model evaluation method that improves the evaluation capabilities of large language models (LLMs) through self-training, eliminating the need for manually labeled data. Starting with unlabeled instructions, iteratively...
What are Self-Taught Evaluators?
Self-Taught Evaluators are a novel model evaluation method that improves the evaluation capabilities of large language models (LLMs) through self-training, without requiring manually labeled data. Starting with unlabeled instructions, it iterative self-improvement generates comparative model outputs. The LLM acts as the judge, generating inference trajectories and final judgments. This process is repeated in each iteration, training the model with the improved predictions. In experiments, Self-Taught Evaluators improve the evaluation accuracy of the Llama3-70B-Instruct-based model from 75.4 to 88.3, reaching 88.7 in majority voting, surpassing commonly used LLM judges such as GPT-4, and performing comparably to top-performing reward models trained with manually labeled data.
Main functions of Self-Taught Evaluators
- Generate comparison model outputStarting with unlabeled instructions, generate model response pairs of varying quality based on prompts.
- Training LLMs as refereesUse LLM to generate inference trajectories and final decisions, and evaluate which response is better.
- Iterative self-improvementIn each iteration, the training data is labeled with the current model's judgments, the model is fine-tuned, and self-improvement is achieved.
- Evaluate model performanceThe accuracy of the model is evaluated on standard evaluation protocols such as RewardBench and compared with human evaluation results.
The technical principle of Self-Taught Evaluators
- initializationAssume access to a large number of human-written user instructions and an initial seed LLM.
- Command SelectionInstructions are classified based on LLM, and a subset of instructions with challenging and balanced distribution is selected.
- Response to ConstructionGenerate preference data for each selected instruction, including two responses (preferred and non-preferred), generated based on prompts, ensuring that the quality of the non-preferred response is lower than that of the preferred response.
- Iterative trainingThis process includes two steps: annotation evaluation and model fine-tuning. First, the current model is used to generate inference trajectories and judgments. If the judgment is correct, the example is added to the training set. Then, the model is fine-tuned using data to provide an updated model for the next iteration.
Project address for Self-Taught Evaluators
- GitHub repository:https://github.com/facebookresearch/RAM/tree/main/projects/self_taught_evaluator
- HuggingFace model library:https://huggingface.co/datasets/facebook/Self-taught-evaluator-DPO-data
- arXiv technical paper:https://arxiv.org/pdf/2408.02666
Application scenarios of self-taught evaluators
- Language model developmentWhen developing new large language models (LLMs), self-taught evaluators assess and optimize the output quality of the model to ensure that the text generated by the model meets the expected standards.
- Automated content evaluationIn the content production field, such as news organizations, publishing industry, or social media platforms, it is used to automate the evaluation of content quality and accuracy, and improve the efficiency of content review.
- Education and academic researchIn the field of education, self-taught evaluators serve as supplementary tools to help assess students' writing assignments or research papers, providing feedback and suggestions for improvement.
- Customer service and technical supportIn customer service, it is used to evaluate the quality of automated response systems, ensuring that responses are both accurate and helpful, thereby improving customer satisfaction.
- Programming and code generationFor scenarios requiring code generation and evaluation, Self-Taught Evaluators can assess the quality of generated code snippets, helping developers improve their code.