WorldVLA - An autoregressive action world model jointly developed by Alibaba DAMO Academy and Zhejiang University

What is WorldVLA?

WorldVLA, a joint development by Alibaba DAMO Academy and Zhejiang University, is an autoregressive action-world model that integrates a visual-language-action (VLA) model with a world model into a single framework. The model predicts future images based on action and image understanding, aiming to learn the fundamental physical laws of the environment to improve action generation. The action model generates subsequent actions based on image observations, aiding visual understanding and, in turn, assisting the world model's visual generation. WorldVLA outperforms independent action and world models, highlighting the synergistic effect between the two. To address the performance degradation issue when generating a series of actions using an autoregressive approach, an attention masking strategy is proposed, selectively masking previous actions when generating the current action, significantly improving performance in action block generation tasks.

Main functions of WorldVLA

• Action generationIt generates subsequent actions based on images and language commands, and supports continuous action planning.
• Image predictionIt predicts future image states based on current images and actions, improving the accuracy of visual prediction.
• Environmental understandingLearn the physical laws of the environment and enhance visual and motor comprehension abilities.
• Bidirectional enhancementThe motion model and the world model complement each other, improving overall performance.

The technical principles of WorldVLA

• Unified frameworkWorldVLA integrates the Visual-Language-Action (VLA) model and the world model into a single framework. It uses three independent encoders (image encoder, text encoder, and action encoder) to encode data from different modalities into tags in a unified vocabulary, enabling cross-modal understanding and generation.
• Autoregressive generationThe model uses an autoregressive approach to generate actions and images. The action model generates actions based on historical images and verbal commands, while the world model predicts future image states based on historical images and actions.
• Attention masking strategyTo address the potential performance degradation of autoregressive models when generating a series of actions, WorldVLA proposes an attention masking strategy. This strategy selectively masks previous actions when generating the current action, reducing error propagation and improving the performance of action block generation.
• Bidirectional enhancementWorldVLA achieves bidirectional enhancement through the interaction of a world model and an action model. The world model helps the action model better understand the physical laws of the environment by predicting future states, while the action model helps the world model more accurately predict future image states by generating actions.
• Training strategyWorldVLA uses a hybrid approach during training, combining action model data and world model data to ensure the model can simultaneously learn action generation and image prediction capabilities. This hybrid training strategy allows the model to perform multiple functions within a single architecture.

WorldVLA's project address

• GitHub repositoryhttps://github.com/alibaba-damo-academy/WorldVLA
• HuggingFace model library: https://huggingface.co/collections/jcenaa/worldvla-685b9df63bdfe8cb67cc71b2
• arXiv technical paper: https://arxiv.org/pdf/2506.21539

Application scenarios of WorldVLA

• Robot Goal-Oriented TaskIt helps robots complete goal-oriented tasks based on visual and verbal instructions, such as moving an object from one location to another.
• Fine manipulation in complex environmentsIn complex environments, such as cluttered desktops or confined spaces, it generates highly adaptable actions to complete precise operations.
• Human-machine collaborative tasksIn human-computer collaboration scenarios, understanding human actions and intentions, generating corresponding collaborative actions, and improving collaboration efficiency.
• Future Scenario Simulation and PredictionPredicting future image states helps robots plan ahead and assess the consequences of their actions, such as road scene prediction in autonomous driving.
• Education and Research PlatformAs a teaching tool and research platform, it helps students and researchers understand and practice the principles of robot control and visual prediction.