WebShaper - An AI training data synthesis system launched by Alibaba Tongyi

What is WebShaper?

WebShaper is an innovative AI training data synthesis system launched by Alibaba Tongyi Labs. Through formal modeling and agent expansion mechanisms, it provides high-quality, scalable data for training AI agents. WebShaper introduces the concept of "Knowledge Projection" (KP) based on set theory for the first time. Through the intersection, union, and recursive operations of KP, it constructs complex problem structures, precisely controlling the inference path and task complexity. WebShaper's Expander agent can start from a simple "seed problem" and gradually expand into complex inference tasks, allowing the AI to "propose its own questions." The training strategy combines Supervised Fine-Tuning (SFT) and GRPO reinforcement learning, enabling the model to perform exceptionally well in complex information retrieval tasks.

WebShaper's main functions

• Formal modelingWebShaper pioneered a formal modeling method for information retrieval (IS) tasks based on set theory. It decomposes complex information retrieval tasks into multiple set operations (such as intersection, union, and recursion) through "Knowledge Projection" (KP). Each KP is a set containing specific entities, and these operations allow for the construction of complex problem structures, precisely controlling the reasoning path and task complexity.
• Agent extension mechanismOne of WebShaper's major innovations is enabling AI to "propose" its own questions. Through the Expander agent, the system starts with a simple "seed problem" and gradually expands it into complex reasoning tasks. The Expander agent invokes tools such as search, summarization, and verification to progressively construct more complex and logically clear questions, and verifies the correctness of the answers. This ensures a clear reasoning chain and a controllable task structure.
• High-quality data generationWebShaper, through formal modeling and agent extension mechanisms, generates training data that is no longer based on guesswork, but rather on controllable, interpretable, and scalable high-quality tasks. It breaks through the boundaries of pre-retrieved data, enabling a wider range of task types, capability activation, and knowledge coverage, while reducing errors and redundant information in data synthesis.
• Agent training strategyWebShaper employs a combined strategy of Supervised Fine-Tuning (SFT) and GRPO reinforcement learning, enabling the AI agent to gradually master reasoning and retrieval capabilities amidst ambiguous and multi-hop information. Training begins with high-quality training trajectories, and a reward mechanism guides the model to perform multi-step reasoning, avoiding shortcuts or guessing.

WebShaper's technical principles

• Formal-driven frameworkWebShaper uses set theory to systematically formalize information retrieval tasks, with the core concept being "Knowledge Projections" (KP). A KP is a set of entities based on a specific relation.
• Knowledge projection operation
  • R-unionUsed to handle uncertain conditions, such as "players who participated in the competition from 2000 to 2010" can be represented by a union operation.
  • IntersectionUsed to handle multiple constraints, such as "players who participated in 2000 and were born in the 1990s".
• Task extension mechanismWebShaper starts with a "seed task" and uses expanders to progressively increase the complexity of the problem. Expanders, based on a formal framework and combined with retrieval and validation tools, extend simple problems into complex ones, ensuring logical consistency and increasing task difficulty.
• Data synthesis and trainingThe generated complex questions are converted into training data, and the model is trained through supervised fine-tuning (SFT) and reinforcement learning (such as the GRPO algorithm) to improve the model's reasoning ability in complex information retrieval tasks.

WebShaper's project address

• Github repositoryhttps://github.com/Alibaba-NLP/WebAgent
• HuggingFace model libraryhttps://huggingface.co/datasets/Alibaba-NLP/WebShaper
• arXiv technical paper: https://arxiv.org/pdf/2507.15061

Application scenarios of WebShaper

• Literature review and analysisWebShaper can help researchers quickly collect and organize relevant literature for interdisciplinary knowledge discovery.
• Market researchWebShaper can be used for market research, competitive analysis, and investment decision support. Business analysts can enable AI systems to automatically collect industry data, analyze market trends, and evaluate competitor strategies.
• Intelligent learning assistantWebShaper can be an intelligent learning assistant, helping students engage in deep learning and research-based learning.
• Life DecisionsWebShaper can be used immediately in scenarios such as travel planning, health inquiries, and life decision-making, providing users with personalized information support.
• Medical Information InquiryWebShaper can help users access medical and health information and provide professional medical advice and health consultations.