AReaL 2.0 - An online reinforcement learning infrastructure for agents developed by Ant Financial and other open-source companies.

What is AReaL 2.0?

AReaL 2.0 is an open-source online reinforcement learning infrastructure for agents, developed by Ant Financial in collaboration with Hong Kong University of Science and Technology and Tsinghua University. It enables deployed agents to continuously evolve from real-world interaction trajectories. AReaL 2.0 transforms production-environment session streams into trainable experience through three pillars: a trajectory data protocol, an enterprise-grade data agent, and an evolution control plane. This eliminates the need to rewrite agents or overhaul business systems. It has been added to the PyTorch Foundation and supports Ascend NPU adaptation.

Main functions of AReaL 2.0

• Online RL closed loopIt automatically transforms the agent's real interaction trajectory into trainable experience, realizing a closed loop from execution to learning.
• Trajectory data collectionThe ATDP protocol records the complete decision-making process step by step, including observations, actions, results, and metadata.
• Enterprise-level data governanceData Proxy is responsible for interception, desensitization, access control, and trajectory persistence to ensure compliance processing is completed before training.
• Evolutionary Control PlaneBased on multi-dimensional signals such as trajectory statistics, failure clusters, and user correction rates, determine the repair entry points such as model updates, memory writes, or tool adjustments.
• microservice architectureThe training, inference, and weight update processes are broken down into independently deployable components such as Gateway, Router, and Data Proxy.

Technical Principles of AReaL 2.0

• Agent Trajectory Data Protocol (ATDP)A learning-oriented agent trajectory protocol breaks down complex tasks into traceable and replayable learning samples. It records agent observations, internal states, action choices, action results, reward feedback, and metadata such as model version, tool version, tenant, and cost, step by step, answering key questions such as "which retrieval or tool call affected the success or failure of the task."
• Agentic Data ProxyThis is the learning data layer deployed at key boundaries such as models, tools, retrieval systems, and human feedback channels. It is responsible for interception, collection, anonymization, access control, trajectory persistence, reward collection, and replay management. Data governance is completed before it enters the training queue, distinguishing between visible fields, training eligibility data, and data used only for debugging and auditing.
• Agent Evolution Control PlaneThe intelligent agent evolution control plane transforms "whether to update and where to update" into a governable systemic decision. Based on trajectory statistics, user correction rates, tool failure clusters, evaluator scores, cost signals, security constraints, and distribution drift, it determines whether evolution should fall at the level of model RL, preference optimization, memory writing, or tool routing adjustment, and supports replay evaluation, regression testing, canary release, and version tracking.
• Online RL microservice runtimeThe existing RL infrastructure is transformed into an online system that handles traffic from Agent services. The Gateway acts as the entry point to receive requests; the Router maintains the binding relationship between sessions and the Data Proxy to ensure the continuity of multi-round interaction context; the Data Proxy manages session state and trajectory; the Agent-Compute Worker performs inference, sampling, and training computations; and the Controller is responsible for scheduling, scaling, and health checks.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use AReaL 2.0

• Visit the project homepageOpen the GitHub repository https://github.com/areal-project/AReaL to obtain the source code and documentation.
• Install infrastructureDeploy microservice components such as Gateway, Router, Data Proxy, and Controller of AREAL 2.0.
• Switch inference entry point: Point the existing Agent's LLM API call backend (base_url/api_key) to the Gateway managed by AREAL 2.0.
• Configure Track ProtocolEnable ATDP to record agent observations, actions, tool calls, and results step by step.
• Deploy data brokerConfigure Data Proxy to perform data interception, data masking, access control, and persistent tracking.
• Start the training closed loopImport the collected real trajectories into the training service, start online RL, and let the agent learn continuously from the interaction.

AReaL 2.0's core advantages

• Low-intrusion accessSimply switch the LLM inference backend; there is no need to rewrite the Agent's planning, tool calls, sandbox, or memory modules.
• Real trajectory driven: Training directly with real online interactive data bridges the gap between offline simulation environments and actual behavior.
• Enterprise-level security governanceData anonymization, access control, and compliance boundaries are completed before training, adapting to multi-tenant and multi-business-line scenarios.
• Explainable and rollbackableEach evolution undergoes replay evaluation, regression testing, and version tracking, allowing for quick rollback if issues arise.
• Rich open source ecosystemIt has joined the PyTorch Foundation, and the community provides Ascend NPU adaptation and LoRA low-computing-power solutions.

AReaL 2.0 project address

• Project official website: https://areal-ai.io/en/intro.html
• GitHub repositoryhttps://github.com/areal-project/AReaL
• arXiv technical paper: https://arxiv.org/pdf/2607.01120

AReaL 2.0 Competitive Product Comparison

Application scenarios of AReaL 2.0