SpatialVLA - A general operational model for spatial embodiment developed by Shanghai AI Lab in collaboration with ShanghaiTech University and others.

What is SpatialVLA?

SpatialVLA is a novel spatial embodied general manipulation model jointly developed by the Shanghai AI Lab, the China Telecom Artificial Intelligence Research Institute, and ShanghaiTech University. Pre-trained on millions of real-world datasets, it endows robots with general 3D spatial understanding capabilities. SpatialVLA fuses 3D spatial information with semantic features using Ego3D position encoding and discretizes continuous actions using an adaptive motion mesh, achieving generalized control across robot platforms. Pre-trained on large-scale real-world robot data, SpatialVLA demonstrates powerful zero-shot generalization and spatial understanding capabilities, excelling in complex environments and multi-task scenarios. Its open-source code and flexible fine-tuning mechanism provide a new technological path for research and applications in the field of robotics.

Main functions of SpatialVLA

• Zero-shot generalization controlIt can perform operations directly in unfamiliar robotic tasks and environments without additional training.
• Efficiently adapt to new scenarios: Fine-tune with a small amount of data to quickly adapt to new robot platforms or tasks.
• Powerful spatial understanding capabilitiesIt can understand complex 3D spatial layouts and perform precise operational tasks, such as object positioning, grasping, and placement.
• Cross-platform versatilityIt supports various robot forms and configurations, enabling universal operation strategies.
• Rapid reasoning and efficient action generationBased on a discretized action space, it improves the model inference speed and is suitable for real-time robot control.

SpatialVLA Technical Principles

• Ego3D Location EncodingThis approach combines depth information with 2D semantic features to construct a robot-centric 3D coordinate system. It eliminates the need for robot-camera calibration, allowing the model to perceive 3D scene structure and adapt to different robot platforms.
• Adaptive Action MeshThe system discretizes continuous robot movements into an adaptive mesh and partitions the motion space based on data distribution. Movements from different robots are aligned using the mesh, enabling cross-platform motion generalization and transfer.
• Spatial Embedding AdaptationDuring the fine-tuning phase, the mesh is re-divided based on the new robot's motion distribution, and the spatial embedding is adjusted. This provides a flexible and efficient robot-specific post-training method to accelerate the model's adaptation to new environments.
• Pre-training and fine-tuningThe model is pre-trained on large-scale real-world robot data to learn general operational strategies. Fine-tuning is then performed on new tasks or robot platforms to further optimize model performance.

SpatialVLA project address

• Project official website:https://spatialvla.github.io/
• GitHub repository:https://github.com/SpatialVLA/SpatialVLA
• HuggingFace model library:https://huggingface.co/IPEC-COMMUNITY/foundation-vision-language-action-model
• arXiv technical paper:https://arxiv.org/pdf/2501.15830

Application scenarios of SpatialVLA

• Industrial manufacturingUsed for automated assembly and parts handling, it can quickly adapt to different production lines and improve production efficiency.
• Logistics warehousingIt can accurately grasp and transport goods, adapt to dynamic environments, and optimize logistics efficiency.
• service industryIt can perform delivery, cleaning, and sorting tasks, understand natural language instructions, and adapt to complex environments.
• Medical assistance: To pass surgical instruments and transport medications, ensuring accurate and safe operation.
• Education and ResearchIt supports the rapid development and testing of new robotic applications, and assists academic research.