VitaBench 2.0 - Meituan LongCat's long-term dynamic agent evaluation benchmark

What is VitaBench 2.0?

VitaBench 2.0 is the first benchmark for evaluating intelligent agents in real-life scenarios using long-term dynamic user modeling, launched by Meituan's LongCat team. It includes 56 simulated users, 819 complex tasks, over 2000 dynamic preferences, and 66 executable tools, with an average interaction span of 1580 days. The system evaluates the personalization and initiative capabilities of large models in long-term interactions.

Main features of VitaBench 2.0

• Simulated user trajectory construction: To build long-term life trajectories for 56 virtual users, covering areas such as food delivery, in-store services, and business travel.
• Dynamic preference evolutionIt embeds over 2,000 preferences, with an average of more than 48 dynamic changes per user.
• Complex task evaluationIt provides 819 executable tasks throughout the user lifecycle.
• Dual Memory Mode ShowdownUnified evaluation of Agentic Memory (actively maintaining archives) and RAG Memory (retrieval of historical fragments).
• Proactive task designThis tests AI's ability to proactively ask questions and make decisions when information is insufficient.

Technical Principles of VitaBench 2.0

• 3D Deconstruction ArchitectureThe system connects user information (profile + preferences + history + tasks) → personalized memory → intelligent agent tasks into a complete closed loop.
• Time scale exposedThe interaction events are exposed to the Agent strictly according to the timeline, so as to truly restore the evolution and drift of user preferences.
• Memory Arena Mechanism: By using an extensible interface, two memory architectures can be compared fairly in the same user scenario to evaluate the real impact of different designs on decision-making.
• Noise signal separationApproximately 20% of the interactions contain irrelevant, exploratory, and proxy noise, testing the model's ability to extract genuine preferences from mixed cues.

How to use VitaBench 2.0

• Cloning repository:implement git clone https://github.com/meituan-longcat/vitabench-2.0.git Obtain the evaluation framework and execution script.
• Download dataPull a standardized dataset from the HuggingFace dataset, containing trajectories of 56 users, 819 tasks, and 66 tools.
• Install dependenciesNavigate to the project directory and run the installation command to configure the Python environment and required libraries.
• Select modeChoose one of the three memory settings—Full Context, Agentic Memory, and RAG Memory—as the benchmark.
• Access ModelFollow the repository documentation to connect your large model or Agent to the evaluation interface, enabling it to receive user events and invoke tools according to the timeline.
• Startup EvaluationRun the evaluation script to allow the Agent to perform standardized tasks sequentially in scenarios such as food delivery, in-store visits, and business travel.
• View reportAnalyze the generated performance report, compare metrics such as Avg@4 and Pass@4 with time decay curves, and identify the model's shortcomings in long-term user modeling.

VitaBench 2.0's core advantages

• Industry firstThe first evaluation benchmark that combines intelligent agent scenarios with a rich user ecosystem and is geared towards long-term dynamic user modeling in real life.
• Ultra-long time spanThe average interaction cycle is 1580 days (maximum 2974 days), truly reflecting the long-term evolution and drift of user preferences.
• Highly realistic user profilesThe 56 virtual users are constructed based on real-world statistical data, covering multiple dimensions such as gender, age, city level, occupation, and marital status.
• Dynamic preference evolutionIt embeds over 2,000 preferences, with each user experiencing more than 48 dynamic changes on average, simulating habit changes in real life.
• Unified Memory Assessment Ecosystem: Build the first long-term intelligent agent evaluation platform in a real-world scenario, and compare the two architectures of Agentic Memory and RAG Memory in a unified manner.

VitaBench 2.0 project address

• Project official websitehttps://vitabench2.github.io/
• GitHub repository: https://github.com/meituan-longcat/vitabench-2.0
• HuggingFace model library: https://huggingface.co/datasets/meituan-longcat/VitaBench-2.0
• arXiv technical paper: https://arxiv.org/pdf/2605.27141

VitaBench 2.0 Competitive Product Comparison

Application scenarios of VitaBench 2.0

• AI Personal Assistant ReviewThe test aims to assess the intelligent assistant's ability to understand and provide personalized services during long-term companionship.
• Intelligent customer service optimization: Evaluate the accuracy of the customer service system's memory and dynamic adaptation of user preferences.
• Lifestyle service platformIt provides a realistic evaluation environment for recommendation algorithms in scenarios such as food delivery, travel, and hotels.
• Memory module development: To help researchers compare the performance of Agentic Memory and RAG Memory in different scenarios.
• Boundary exploration of long context modelsTo test the limits of large models in ultra-long time series and high-noise scenarios.