SWEET-RL - A multi-round reinforcement learning framework launched by Meta
SWEET-RL is a multi-turn reinforcement learning framework from Meta, specifically designed for training large language model (LLM) agents for collaborative reasoning tasks. SWEET-RL optimizes the "critic"... based on additional information during training (such as a reference solution).
What is SWEET-RL?
SWEET-RL is a multi-turn reinforcement learning framework from Meta, specifically designed for training large language model (LLM) agents for collaborative reasoning tasks. SWEET-RL optimizes the "critic" model based on additional training information (such as a reference solution), providing rewards for each step to help the "actor" model better allocate credits and optimize policies. SWEET-RL performs exceptionally well on ColBench benchmarks, achieving a 6% improvement in success rate and win rate on backend programming and frontend design tasks compared to other state-of-the-art algorithms. This makes the performance of the Llama-3.1-8B model comparable to or even surpassing top models like GPT-4o.
Main functions of SWEET-RL
- Optimize multi-turn interaction tasksSWEET-RL is specifically optimized for complex tasks that require multiple rounds of interaction, such as backend programming and frontend design.
- Effective allocation of creditBased on additional information introduced during training (such as a reference solution), rewards are provided for each step, accurately evaluating the value of each action and solving the problem of credit allocation in multi-round tasks.
- Supports multiple task typesIt supports handling complex front-end design tasks, demonstrating its versatility and adaptability across different task types.
SWEET-RL Technical Principles
- Additional information during trainingSWEET-RL optimizes the "critic" model based on additional information during training, such as a reference solution. The critic model provides a reward for each step, helping the "actor" model to better allocate credit.
- Bradley-Terry's goalSWEET-RL directly trains the advantage function using the Bradley-Terry objective function, which evaluates the effectiveness of each action in the current state. This avoids pre-training the value function to predict the expected utility of the current state and action, and better aligns with pre-trained LLMs.
- Asymmetric information structureBased on an asymmetric actor-critic structure, where the critic model accesses additional information during training, and the actor model accesses the interaction history. This allows the critic to more accurately evaluate the value of actions, and the actor to optimize the strategy based on the evaluation.
- Parameterized advantage functionThe advantage function is parameterized as the average log probability of each action, and training is performed based on the trajectory-level Bradley-Terry objective. This parameterization method is more consistent with the pre-training objective of LLM, improving the model's generalization ability.
SWEET-RL project address
- GitHub repository:https://github.com/facebookresearch/sweet_rl
- HuggingFace model library:https://huggingface.co/datasets/facebook/collaborative_agent_bench
- arXiv technical paper:https://arxiv.org/pdf/2503.15478
Application Scenarios of SWEET-RL
- Text proofreadingIt helps authors and editors quickly correct typos and sensitive content in articles.
- Social media moderationEnsure that content posted on social media is compliant and protect the reputation of individuals or businesses.
- Advertising complianceReview advertising copy to avoid legal and market risks caused by content errors.
- Academic PublishingTo ensure the accuracy and rigor of teaching materials and academic works.
- Multimedia content detectionReview videos, audio, and images to ensure that multimedia content is legal and compliant.