VSI-Bench - A visual spatial intelligence benchmark suite developed by Li Feifei and Xie Saining's team.
VSI-Bench (Visual-Spatial Intelligence Benchmark) is a visual-spatial intelligence benchmark set developed by Fei-Fei Li, Sai-Ning Xie, and their research team. The researchers constructed it to evaluate large-scale multimodal language models...
What is VSI-Bench?
VSI-Bench (Visual-Spatial Intelligence Benchmark) is a visual-spatial intelligence benchmark suite developed by Fei-Fei Li, Sai-Ning Xie, and their research team. Researchers constructed it to evaluate the spatial cognition and understanding capabilities of multimodal large language models (MLLMs). VSI-Bench contains over 5000 question-answer pairs, covering nearly 290 real-world indoor scene videos, encompassing various environments such as residences, offices, and factories. VSI-Bench tasks are divided into configuration tasks (such as object counting and relative distance), measurement estimation (such as object size and room size), and spatiotemporal tasks (such as object appearance order), systematically testing and improving the performance of MLLMs in visual-spatial intelligence.
Main functions of VSI-Bench
- Assessing visual spatial intelligence: Quantitatively evaluate the visual spatial intelligence of multimodal large language models (MLLMs), including their ability to perceive, understand, and remember spatial relationships.
- BenchmarkingIt provides a standardized test set containing more than 5,000 question-answer pairs for benchmarking and comparing the performance of different MLLMs on visual spatial tasks.
- Task diversityIt includes configuration tasks (object counting, relative distance, relative direction, route planning), measurement estimation (object size, room size, absolute distance), and spatiotemporal tasks (object appearance order), comprehensively covering multiple aspects of visual spatial intelligence.
- Video UnderstandingBased on video input, VSI-Bench tests MLLMs' understanding of continuous, temporal input, which is closer to how humans observe the world than static images.
- Data quality and controlHuman review ensures data quality, eliminates ambiguity and incorrect labeling, and improves the reliability of test results.
VSI-Bench Technical Principles
- Dataset ConstructionBased on multiple public indoor 3D scene reconstruction datasets (such as ScanNet, ScanNet++, and ARKitScenes), the datasets provide high-fidelity video scans and object-level 3D annotations.
- Question-answer pair generationThe system automatically generates question-answer pairs based on metadata (such as object categories and bounding boxes) and question templates in the dataset, while manually annotating the route planning task.
- Quality controlImplement a manual review process to ensure that issues are clear and unambiguous, and trace and correct errors or ambiguities.
- Model EvaluationThe study evaluated various video-supported MLLMs in a zero-sample setting, using default prompts for testing, and employing exact matching and fuzzy matching as the primary evaluation metrics.
- Performance indicatorsFor multiple choice (MCA) tasks, accuracy (ACC) is used; for numerical answer (NA) tasks, a new metric is introduced—mean relative accuracy (MRA).
- Cognitive Map GenerationThis prompts MLLMs to predict the center position of objects in a video, generate cognitive maps, and evaluate the model's internal spatial representation and memory capabilities.
VSI-Bench project address
- Project official website:vision-x-nyu.github.io/thinking-in-space
- GitHub repository:https://github.com/vision-x-nyu/thinking-in-space
- HuggingFace model library:https://huggingface.co/datasets/nyu-visionx/VSI-Bench
- arXiv technical paper:https://arxiv.org/pdf/2412.14171
VSI-Bench application scenarios
- Robot navigation and interactionIn the field of robotics, evaluating and training MLLMs can help robots better understand spatial layouts and improve their navigation and obstacle avoidance capabilities in unknown environments.
- Augmented Reality (AR) / Virtual Reality (VR)In AR/VR applications, it helps MLLMs better understand the user's physical spatial environment and provides a more natural interactive experience.
- self-driving cars: Autonomous driving systems require precise spatial understanding to handle complex traffic scenarios, and are used in the development and testing of vision-space intelligence systems that support autonomous driving.
- Smart home systemSmart home systems need to understand the layout of living spaces and the location of objects. VSI-Bench helps train MLLMs to better serve smart home control and automation.
- Interior design and architectural planningTo evaluate the application of MLLMs in spatial planning and design, and to help the system provide more reasonable layout suggestions.