AB
AB
AiBoss

All Categories

project

RAGEN - An open-source reinforcement learning framework for training large-scale model inference agents.

RAGEN is an open-source reinforcement learning framework for training large language model (LLM) inference agents in interactive, stochastic environments. It is based on StarPO (State-Thinking-Action-Reward Policy Optimization)...

What is RAGEN?

RAGEN is an open-source reinforcement learning framework for training large language model (LLM) inference agents in interactive, stochastic environments. Based on the StarPO (State-Thinking-Action-Reward Policy Optimization) framework, it optimizes the entire trajectory through multiple rounds of interaction, supporting various optimization strategies such as PPO and GRPO. RAGEN formalizes the agent-environment interaction through MDP and introduces a progressive reward normalization strategy, effectively addressing the instability in multi-round reinforcement learning. RAGEN's code structure is optimized, divided into three modules: environment manager, context manager, and agent, facilitating extensibility and experimentation. It supports multiple environments, such as Sokoban and FrozenLake, demonstrating good generalization capabilities.

RAGEN's main functions

  • Multi-round interaction and trajectory optimizationRAGEN, through its StarPO (State-Thinking-Actions-Reward Policy Optimization) framework, formalizes the interaction between the agent and the environment as a Markov Decision Process (MDP), optimizing the entire interaction trajectory, not just single actions. This full-trajectory optimization strategy helps the agent make more rational decisions in complex environments.
  • Reinforcement learning algorithm supportRAGEN supports a variety of reinforcement learning algorithms, including PPO, GRPO, and BRPO, providing researchers with flexible algorithm choices.
  • Easily expandable environment supportRAGEN supports various environments, including Sokoban and Frozen Lake, and provides an interface for adding custom environments, making it convenient for researchers to conduct experiments.
  • Stability and efficiency improvementRAGEN effectively improves the stability and efficiency of training through techniques such as variance-based trajectory filtering, the introduction of "criticisms," and decoupling pruning.

RAGEN's technical principles

  • MDP formalizationRAGEN formalizes the interaction between the agent and the environment as a Markov Decision Process (MDP), where states and actions are sequences of tokens. It supports LLM for reasoning about environmental dynamics.
  • StarPO frameworkThe framework is trained through two alternating phases:
    • Rollout phaseGiven an initial state, LLM generates multiple inference-guided interactive trajectories, receiving the trajectory history and generating actions at each step.
    • Update phaseAfter generating the trajectory, importance sampling is used to optimize the expected reward of the entire trajectory. This is not a single-step optimization, but rather a long-term inference.
  • Optimization strategyStarPO supports a variety of reinforcement learning algorithms, such as PPO (Proximal Policy Optimization) and GRPO (Normalized Reward Policy Optimization), to adapt to different training needs.
  • Progressive reward normalization strategyTo address the instability in multi-round training, RAGEN introduces strategies such as uncertainty-based filtering, removal of KL penalties, and asymmetric PPO pruning.
  • Modular designRAGEN employs a modular architecture, including an environment state manager, a context manager, and an agent, which facilitates expansion and customization.

RAGEN's project address

Application scenarios of RAGEN

  • Intelligent dialogue systemRAGEN can be used to train dialogue systems to have better reasoning abilities in interactions with users, and to provide more natural and accurate answers.
  • Game AIIn complex and dynamic game environments, RAGEN can help agents plan and execute strategies effectively.
  • Automated reasoningRAGEN can be applied to automated reasoning scenarios such as solving mathematical problems and programming tasks, improving the system's problem-solving capabilities.
  • Enterprise knowledge managementRAGEN can be used as an internal document assistant for enterprises, locating information from company wikis and meeting minutes to generate project reports or meeting summaries.
  • Legal ConsultationIn the legal field, RAGEN can match relevant legal provisions and precedents, explaining legal risks in plain language.
  • Content creationRAGEN can be used for scenarios such as writing technical blogs and generating news reports. By searching GitHub code examples, technical documentation, etc., RAGEN can integrate information to output structured tutorials.
Back