ArenaRL - A comparative open-domain reinforcement learning method between Tongyi and Gaode Open Source

What is ArenaRL?

ArenaRL is a contrastive reinforcement learning method for open-domain agents, open-sourced by the Tongyi DeepResearch team in collaboration with AutoNavi. ArenaRL introduces a tournament mechanism, transforming traditional absolute scoring into relative ranking within groups. Using an efficient seeded single-elimination topology, it keeps computational complexity at a linear level, effectively solving the discrimination collapse problem caused by the lack of a standard answer in open-domain tasks. ArenaRL performs excellently in academic benchmark tests and has been successfully implemented in real-world business scenarios at AutoNavi, significantly improving the planning and execution capabilities of complex tasks.

ArenaRL's main functions

• Optimize open domain task performanceArenaRL addresses the bottleneck of traditional reinforcement learning in open-domain tasks caused by the lack of standard answers through contrastive reinforcement learning, helping agents find better solutions in complex tasks.
• Improve training efficiencyArenaRL uses a seeded single-elimination tournament mechanism to keep the computational complexity within a certain range.O(N) Linear levels achieve the best balance between training efficiency and effectiveness.
• Enhance reasoning and planning abilitiesIntroducing a process-aware evaluation mechanism can assess the final result and examine the accuracy of the thought chain logic and tool calls, thereby improving the agent's reasoning and planning capabilities.
• Supports diverse application scenariosIt performs well in various scenarios such as complex travel planning, in-depth information retrieval, and general writing, demonstrating strong task generalization capabilities.

ArenaRL's technical principles

• From absolute scoring to relative rankingTraditional reinforcement learning relies on reward models to assign absolute scalar scores to each generated trajectory. This approach is prone to discriminative collapse in open-domain tasks. ArenaRL innovatively introduces a relative ranking mechanism, refactoring reward modeling into an intra-group relative ranking problem through pairwise comparisons, thus avoiding the limitations of absolute scoring.
• Tournament Mechanism and Seeded Single-Elimination TournamentArenaRL employs a tournament mechanism, where agents generate a set of candidate solutions for the same instruction, constructing a miniature "arena." Through a seeded single-elimination topology, ArenaRL strictly controls computational complexity. O(N) While maintaining a linear level, the advantage estimation accuracy can closely approximate that of a full round-robin tournament, achieving the best balance between training efficiency and effectiveness.
• Process-aware evaluation mechanismArenaRL can assess the quality of the final result, examining in depth the logical rigor of the CoT (Concept of Thought) and the accuracy of tool calls. This process-aware evaluation mechanism helps agents better weigh multidimensional constraints in complex tasks, improving reasoning and planning capabilities.
• Two-way scoring agreementTo eliminate positional bias when large models act as judges, ArenaRL employs a two-way scoring protocol. During each comparison, the system swaps the order of the two candidate solutions before scoring, ensuring the fairness and fine-grained nature of the evaluation results.
• Open source data and training frameworksArenaRL has open-sourced a complete training framework and end-to-end evaluation benchmarks, including two major benchmark datasets: Open-Travel and Open-DeepResearch. This provides developers with abundant training and testing data, lowers the barrier to entry, and promotes research and application of open-domain intelligent agents.

ArenaRL's project address

• Project official website: https://tongyi-agent.github.io/zh/blog/arenarl/
• GitHub repositoryhttps://github.com/Alibaba-NLP/qqr
• HuggingFace model libraryhttps://huggingface.co/papers/2601.06487
• arXiv technical paper: https://arxiv.org/pdf/2601.06487

ArenaRL Application Scenarios

• Complex travel planningArenaRL can generate multiple candidate routes for users and filter them by comparison to find the optimal route that best meets their fuzzy needs (such as fewer people, shade, and suitability for strollers).
• Long text generation and information retrievalIn long text generation tasks, ArenaRL can significantly improve the command compliance and usability of the generated content, avoiding quality degradation caused by length deviations.
• Multi-tool collaborative tasksArenaRL ensures efficient completion of complex instructions in multi-tool collaborative tasks and avoids unreasonable calls by evaluating the accuracy of thought chain logic and tool calls.
• Personalized recommendationsIn personalized recommendation scenarios, ArenaRL selects the option that best matches the user's vague and personalized needs (such as suitable for dating or having a river-view terrace) from multiple candidate options.
• Open Domain QuestionsIn open-domain question answering, ArenaRL uses relative comparison to select the most reasonable and useful answer from multiple candidate answers, thereby improving the quality of responses.