WBench - Meituan's interactive video world model multi-round evaluation benchmark

What is WBench?

WBench, launched by Meituan's LongCat team, is the first systematic multi-round evaluation benchmark for interactive video world models. It includes 289 test cases and 1058 interaction rounds, covering 6 scene categories including nature, city, and fantasy, and 7 art styles. The benchmark uses a unified interactive interface to perform a comprehensive analysis of 20 cutting-edge models, accurately identifying the boundaries of their capabilities from passive viewing to active interaction, and revealing the core shortcomings of current world models in areas such as multi-round interaction, navigation control, and perspective switching.

WBench's main functions

• Multidimensional world definitionSupports 6 scene types (nature, city, indoor, work area, fantasy, sports), 7 art styles (realism, animation, cartoon, oil painting, ink painting, flat design, sketch) and 3 perspectives including first person/third person/embodied first person.
• Four types of core interactionsIt offers four interaction methods: navigation, main action, event editing, and perspective switching.
• Unified Interaction InterfaceAction Router maps text commands, 6DoF coordinates, and keyboard actions into a standardized input, adapting to the native interfaces of different models.
• Five-dimensional assessment kitThe scoring is based on five dimensions: video quality, setting compliance, interaction compliance, consistency, and physical realism, including core metrics such as NavScore and Gated Spatial Consistency.
• Multiple rounds of long-term assessmentSupports continuous multi-round interactive testing to measure the performance degradation of the model after long-term interaction.

WBench's technical principles

• Four core elements frameworkWBench is designed following a framework of world definition + instruction set + unified interaction interface + evaluation suite. The world definition is responsible for building diverse testing environments; the instruction set provides standardized action descriptions; the unified interaction interface converts heterogeneous inputs into signals that the model can understand through the Action Router; and the evaluation suite outputs quantitative indicators from five dimensions, forming a complete evaluation loop.
• Decoupling of Scene and Interaction DesignThe design adopts a concept that separates the stage from the script: the scenes and interactions are configured independently and can be freely combined. This decoupling allows the evaluation to accurately pinpoint the root cause of the problem.
• Multi-round autoregressive feedback mechanismThe evaluation process employs an autoregressive approach: the output frames of each round serve as the input for the next round, simulating real-world interaction scenarios. In this way, WBench can detect the cumulative error effect of the model in long-term interactions, such as a 33-point average decrease in navigation capability after multiple rounds.

How to use WBench

• Cloning repositoryAccess the GitHub repository and clone the WBench code to your local machine, then install the Python dependencies.
• Download datasetThe evaluation dataset containing 289 test cases and 1058 interaction rounds was pulled from HuggingFace.
• Access ModelThe Action Router connects the model under test to WBench's unified interactive interface, adapting to text or action input protocols.
• Define the scenarioIn the configuration file, select the scene type, art style, main object, and viewing angle.
• Select InteractionTest actions by combining four types of interactions: navigation, main actions, event editing, and perspective switching.
• Set dimensionsEnable five-dimensional evaluation indicators: video quality, setting compliance, interaction compliance, consistency, and physical authenticity.
• Single round test: Run the initial frame generation and single-round interaction to obtain the model's basic ability score.
• Multiple rounds of testing: Initiate autoregressive mode, use the output of the previous round as the input of the next round, and measure long-term performance degradation.
• Batch comparisonConfigure multiple models to run the same test cases in parallel to generate cross-sectional comparison data.
• View reportAfter the evaluation is completed, a radar chart and score table are automatically generated to intuitively display the strength and weakness distribution of each dimension.

• World Model DevelopmentProvides standardized evaluation tools for video world models (such as Genie, Kling, Wan, etc.) to guide model iteration and optimization.
• Navigation and Embodied Intelligence AssessmentIt is specifically designed to evaluate the navigation and control capabilities of models in virtual environments, and is applicable to embodied intelligence fields such as robotics and autonomous driving.
• Multi-round interactive product testingIt helps evaluate the long-term interactive experience of AI video/game products, such as virtual world exploration and AI game engines.
• Academic research benchmarksAs a fundamental evaluation framework for academic research on world models and interactive video generation, it promotes standardization in the field.
• Open source vs. closed source model selectionIt provides objective data for businesses and developers to help them select model solutions suitable for specific interactive tasks (such as navigation/editing).