AB
AB
AiBoss

All Categories

project

WebRL - A self-evolving online course reinforcement learning framework jointly launched by Tsinghua University and Zhipu AI.

WebRL is a self-evolving online course reinforcement learning framework jointly launched by Tsinghua University and Zhipu AI. It trains high-performance network agents using open large language models (LLMs). WebRL dynamically generates tasks, supervises results, and provides rewards...

What is WebRL?

WebRL is a self-evolving online course reinforcement learning framework jointly developed by Tsinghua University and Zhipu AI. It trains high-performance network agents using open large language models (LLMs). WebRL dynamically generates tasks, uses an outcome-supervised reward model (ORM) to evaluate task success, and employs adaptive reinforcement learning policies to address challenges such as scarce training tasks, sparse feedback signals, and policy distribution drift in online learning. WebRL significantly improves the success rate of models like Llama-3.1 and GLM-4 on the WebArena-Lite benchmark, surpassing proprietary LLM APIs and previously trained network agents, demonstrating its effectiveness in enhancing the task capabilities of open-source LLMs.

Main functions of WebRL

  • Self-Evolution Course LearningWebRL can generate new tasks from failed attempts, dynamically adjust the difficulty and complexity of the tasks, and adapt to the agent's current skill level.
  • Outcome-based supervised reward model (ORM)WebRL trains an ORM to evaluate the success of a task, providing binary reward signals (1 for success, 0 for failure) to guide the agent's learning process.
  • Adaptive reinforcement learning strategyA policy update algorithm based on KL divergence constraints is proposed. WebRL limits the distribution drift during the policy update process, ensuring that the agent does not deviate too far from existing knowledge when learning new tasks.
  • Experience replay bufferWebRL uses an experience replay buffer to retain previous successful experiences, mitigating the risk of catastrophic forgetting and reusing experience during training.
  • Continuous performance improvementWebRL is based on iterative self-evolution, enabling agents to continuously and consistently improve performance in online environments.

The technical principles of WebRL

  • Problem StatementWebRL models network tasks as Markov decision processes (MDPs) with finite horizons, defining states, actions, rewards, and transition probabilities.
  • ORM TrainingBased on the trained LLM as the ORM, it automatically evaluates whether the agent's execution trajectory has successfully completed the task and provides feedback signals.
  • reinforcement learningIn online network environments, WebRL dynamically generates tasks using a self-evolving course learning strategy and uses a KL constraint policy update algorithm to prevent large drifts in policy distribution.
  • Experience replayBased on an experience replay buffer, previous knowledge is preserved, mitigating the risk of catastrophic forgetting. By storing successful trajectories, the challenge of accurately estimating intermediate states of erroneous trajectories is avoided.
  • Self-evolutionary learning strategiesWebRL implements a two-step process of generation and filtering, generating increasingly challenging tasks while still being suitable for the agent's current capabilities, and creating new instructions based on in-breadth evolving technology.
  • Strategy UpdateWebRL considers the KL divergence between the old and new policies when updating policies, ensuring a smooth transition of policies and avoiding performance degradation caused by policy updates.

WebRL project address

WebRL Application Scenarios

  • Web browsing automationWebRL automatically completes web browsing tasks, such as information retrieval, form filling, and online shopping, based on trained agents.
  • Network data extractionIn scenarios where specific data (such as prices, reviews, and news articles) needs to be extracted from web pages, WebRL helps automate the data extraction process.
  • Customer service automationIn the field of customer service, chatbots are used to solve user problems or complete transactions through web page interactions.
  • Network Content ManagementFor website administrators who need to manage large amounts of web content, WebRL automates content updates, publishing, and maintenance tasks.
  • e-commerceOn e-commerce platforms, it helps automate order processing, inventory management, and customer interaction.
Back