WebSailor - An open-source network intelligent agent from Alibaba Tongyi Labs

What is WebSailor?

WebSailor is an open-source network agent from Alibaba's Tongyi Lab, focusing on complex information retrieval and inference tasks. Through innovative data synthesis methods (such as SailorFog-QA) and training techniques (such as rejection sampling fine-tuning and the DUPO algorithm), it excels in challenging tasks, surpassing many well-known models in benchmarks such as BrowseComp and topping the open-source network agent leaderboard. WebSailor's inference reconstruction technology efficiently handles complex tasks, generating concise and accurate inference chains. It performs exceptionally well in complex scenarios and demonstrates strong generalization capabilities in simple tasks.

WebSailor's main functions

• Complex task data synthesisWebSailor uses the SailorFog-QA method to generate complex task data with high uncertainty, simulating complex information search scenarios in the real world.
• Multi-turn tool calls and reasoning refactoringWith the help of open-source inference models, WebSailor can make multiple rounds of tool calls, refactor the inference process, and efficiently handle complex problems.
• Reinforcement learning algorithmsThe DUPO algorithm is adopted, and the training efficiency is optimized through a dynamic sampling strategy, which significantly improves the model's decision-making ability.
• Information retrieval and analysisWebSailor can proactively search and access multiple web pages, analyze the relationships between information, and provide complete and accurate answers.

WebSailor's technical principles

• Data synthesis (SailorFog-QA)WebSailor first generates complex task data with high uncertainty using the SailorFog-QA method. It then employs a "knowledge graph random walk" technique, selecting less common entities from knowledge bases such as Wikipedia as starting points to randomly expand the knowledge graph and construct a complex nonlinear relationship network. Furthermore, it increases the initial uncertainty of the task by fuzzifying the problem description (e.g., changing specific years to time periods, or hiding some information).
• Model training (RFT cold start)During the training phase, WebSailor uses Rejection Sample Fine-Tuning (RFT) for a cold start. RFT initially aligns the model by selecting high-quality solution trajectories, helping the model master basic inference and tool-calling habits.
• Reinforcement learning (DUPO algorithm)After completing the cold start, WebSailor enters the reinforcement learning phase, employing the DUPO (Repetitive Sampling Optimization) algorithm. The core of DUPO lies in its dynamic sampling strategy: filtering out overly simple samples before training and repeatedly sampling challenging trajectories during training. This significantly improves training efficiency, enabling the model to iterate rapidly on complex tasks.

WebSailor project address

• Github repositoryhttps://github.com/Alibaba-NLP/WebAgent

WebSailor Application Scenarios

• Complex Information RetrievalWebSailor can handle fuzzy and complex queries, quickly locating and verifying answers from massive amounts of information through multi-step reasoning and cross-validation.
• Multi-jump question and answer taskIn multi-hop question-answering scenarios, WebSailor can break down complex questions and find answers by using multiple rounds of tool calls and reasoning.
• Scientific Research and Data AnalysisWebSailor can help researchers and analysts quickly sift through complex networks of information, synthesize information from multiple sources, and provide complete and accurate conclusions.
• Training and optimization of high-difficulty tasksThe training dataset SailorFog-QA for WebSailor simulates complex real-world information search scenarios. Through the construction of high uncertainty and complex relationship networks, it can handle complex tasks without a pre-defined solution path.