VitaBench - Meituan's large-scale agent evaluation benchmark

What is VitaBench?

VitaBench is a benchmark for evaluating large-scale intelligent agents for complex problems, released by Meituan's LongCat team. Using high-frequency life scenarios such as food delivery, restaurant dining, and travel as examples, it constructs an interactive evaluation environment containing 66 tools and designs comprehensive cross-scenario tasks. It measures agent performance across three dimensions: deep reasoning, tool usage, and user interaction. For the first time, it quantitatively decomposes agent tasks, builds a large-scale real-world environment database, introduces a real user simulator, and achieves fine-grained behavior coverage through an atomic evaluation criterion (Rubric).

VitaBench's main functions

• Building a complex task evaluation environmentUsing high-frequency life scenarios such as food delivery, restaurant dining, and travel as carriers, an interactive evaluation environment containing 66 tools is constructed, and cross-scenario comprehensive tasks are designed to simulate complex needs in real-world scenarios.
• Quantifying the complexity dimension of tasksThe complexity of tasks is quantified from three dimensions: deep reasoning, tool usage, and user interaction. Reasoning complexity is measured by indicators such as the size of the observation space, partial observability, and the number of reasoning points. Tool complexity is distinguished by single-scenario tasks and cross-scenario tasks. A real user simulator is introduced to measure interaction complexity.
• Achieve fine-grained evaluationDrawing on the latest research, the task objective is broken down into a set of atomized evaluation criteria (Rubric). By scanning the entire dialogue trajectory with overlapping sliding windows, the task completion is judged by a strict "all or nothing" standard, achieving more comprehensive and fine-grained behavioral coverage.
• Provide open source resourcesVitaBench is now fully open source, including a project homepage, paper links, code repositories, and datasets, providing researchers and developers with abundant resources to promote the research and development and implementation of intelligent agents in real-life scenarios.

VitaBench's technical principles

• Multidimensional complexity constructionComplex tasks are constructed through three dimensions: deep reasoning, tool usage, and user interaction, simulating the complexity of tasks in real-life scenarios.
• Real-world database: Construct a large-scale real-world environment database to provide agents with partially observable environments in order to measure their reasoning ability in complex environments.
• User SimulatorIntroducing a real user simulator to simulate the behavior and preferences of different users, enabling the intelligent agent to adapt to diverse user behaviors in multi-turn dialogues.
• Atomization evaluation criteriaThe task objective is broken down into a set of atomic evaluation criteria (Rubric), and fine-grained evaluation of agent behavior is achieved by scanning the dialogue trajectory through a sliding window.
• Cross-scenario task designDesign cross-scenario integrated tasks to examine the intelligent agent's ability to switch and execute between multiple scenarios and integrate information, and evaluate its overall performance in real-life scenarios.

VitaBench project address

• Project official websitehttps://vitabench.github.io
• Github repositoryhttps://github.com/meituan-longcat/vitabench
• arXiv technical paperhttps://arxiv.org/abs/2509.26490
• HuggingFace dataset: https://huggingface.co/datasets/meituan-longcat/VitaBench

VitaBench application scenarios

• Food deliverySimulate the complex needs of users when ordering takeout, such as choosing restaurants and dishes based on personal preferences, budget, and time constraints, and evaluate the ability of the agent to understand user needs, recommend suitable options, and complete the order in multi-turn dialogues.
• Dining at a restaurantIt covers the entire process from finding a restaurant and making a reservation to ordering food and paying the bill, and examines the intelligent agent's reasoning and tool usage capabilities in complex scenarios, such as recommending suitable restaurants based on user needs and handling tasks such as reservations and menu searches.
• TravelIt involves multiple aspects such as travel planning, transportation booking, and attraction recommendations, and evaluates the comprehensive performance of intelligent agents in cross-scenario tasks, such as integrating different tools and information resources to provide users with personalized travel solutions.
• Intelligent agent research and evaluationIt provides standardized evaluation benchmarks for researchers and developers, helps to assess and optimize the performance of intelligent agents in complex tasks, and promotes the development and application of intelligent agent technology.
• Human-computer interaction researchThis study investigates the interaction patterns between intelligent agents and users through real user simulators and multi-turn dialogue tasks, aiming to improve the agent's capabilities in natural language understanding and dialogue management.