project
WorldScore - A unified evaluation benchmark for world generative models from Stanford University.
WorldScore is a unified evaluation benchmark for world generation models proposed by Stanford University. It decomposes world generation into a series of next-scene generation tasks, achieving different levels of performance through explicit camera trajectory-based layout specifications...
What is WorldScore?
WorldScore is a unified evaluation benchmark for world generation models proposed by Stanford University. It decomposes world generation into a series of next-scene generation tasks, achieving unified evaluation of different methods through explicit camera trajectory-based layout specifications. WorldScore evaluates three key aspects of generated worlds: controllability, quality, and dynamism. The benchmark includes a carefully curated dataset covering 3000 test samples, encompassing diverse worlds—static and dynamic, indoor and outdoor, realistic and stylized.
Main functions of WorldScore
- Unified assessment frameworkWorldScore provides a unified evaluation framework for measuring the performance of different world generation models. It decomposes the world generation task into a series of next-stage scene generation tasks and achieves unified evaluation of different methods through explicit camera trajectory-based layout specifications.
- Evaluation DimensionsThe generated world is evaluated from three key aspects: controllability, quality, and dynamism.
- Multi-scene generationWorldScore is the only benchmark that supports multi-scene generation and can evaluate the performance of a model when generating continuous scenes.
- UnityIt can uniformly evaluate 3D, 4D, image-to-video (I2V), and text-to-video (T2V) models, providing a comprehensive evaluation framework.
- Long sequence supportIt supports generating multiple scenarios and evaluating the model's performance in long sequence generation tasks.
- Image conditionsIt supports image-based conditional generation, suitable for image-to-video generation tasks.
- Multiple stylesIt contains data with multiple visual styles, which can be used to evaluate the model's ability to generate data under different styles.
- Camera control: Evaluate the model's ability to follow camera trajectories to ensure that the generated scene conforms to the specified camera motion.
- 3D consistency: Evaluate the stability of the scene's geometry to ensure that the generated 3D scene remains consistent across different viewpoints.
WorldScore's Technical Principles
- Diverse datasetsThe WorldScore dataset contains multimedia data with both dynamic and static configurations, suitable for image-to-video and image-to-3D tasks.
- Dynamic configurationIt includes fields such as image, visual motion, visual style, motion type, style, camera path, object, and cues.
- Static configurationIt includes fields such as image, visual motion, visual style, scene type, category, style, camera path, content list, and tooltip list.
- Dataset sizeThe dataset is divided into a training set and a test set, with 1000 samples in the dynamic configuration and 2000 samples in the static configuration.
- Layout specifications based on camera trajectoryA unified evaluation of different methods can be achieved through clear layout specifications based on camera trajectories.
- Multimodal data supportIt supports multiple modalities of data, including images, videos, and 3D models, making it suitable for multimodal content generation tasks.
WorldScore's project address
- Project official website:https://haoyi-duan.github.io/WorldScore/
- Github repository:https://github.com/haoyi-duan/WorldScore
- arXiv technical paper:https://arxiv.org/pdf/2504.00983
- HuggingFace dataset:https://huggingface.co/datasets/Howieeeee/WorldScore
WorldScore benchmark comparison
WorldScore differs from other existing benchmarks in several aspects, as detailed below:
| Benchmarking | Number of examples | Multiple scenarios | Unity | Long sequences | Image conditions | Multiple styles | Camera control | 3D consistency |
|---|---|---|---|---|---|---|---|---|
| TC-Bench | 150 | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| EvalCrafter | 700 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| FETV | 619 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| VBench | 800 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| T2V-CompBench | 700 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Meng et al. | 160 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Wang et al. | 423 | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| ChronoMagic-Bench | 1649 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| WorldModelBench | 350 | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| WorldScore | 3000 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Application scenarios of WorldScore
- Image to video generationIt generates high-quality video content that can be applied to video production, animation design, and other fields.
- Image to 3D generationConvert 2D images into 3D models for use in virtual reality, augmented reality, and 3D modeling applications.
- Dataset supportThe dataset contains multimedia data with both dynamic and static configurations, suitable for a variety of tasks, and helps researchers optimize and improve models.
- Research and DevelopmentThe WorldScore dataset provides researchers with a standardized testing platform for developing and validating new 3D/4D scene generation algorithms.
- Autonomous driving scenario generationBy generating realistic 3D scenes for training and testing of autonomous driving systems, it helps improve the safety and reliability of autonomous driving systems.