project
Lingbot-VA - An open-source video and action world model from Ant Lingbo Technology.
LingBot-VA is the world's first causal video-action world model for general robot control, open-sourced by AntLingbo Technology. The model unifies video world modeling and policy learning into an autoregressive framework, enabling the robot to simultaneously predict...
What is lingbot-VA?
LingBot-VA is the world's first causal video-action world model for general robot control, open-sourced by Antminer Technology. The model unifies video world modeling and policy learning into an autoregressive framework, enabling robots to simultaneously predict future states and execute precise closed-loop control. The model can learn new skills with only 30-50 real-world demonstrations and significantly outperforms mainstream benchmark models in long-term tasks, data-efficient post-training, and generalization capabilities.
Main functions of lingbot-VA
- Unified Video-Motion ModelingIt integrates visual dynamic prediction and action execution into a single framework to achieve closed-loop control of "imagining and acting simultaneously".
- Long-term task executionThe model excels at handling complex tasks that require multi-step planning (such as preparing breakfast or unpacking packages), possesses long-term memory capabilities, and is not misled by loop states.
- High-efficiency post-trainingNew skills can be learned with only 30-50 real demonstrations, with a success rate approximately 20% higher than the baseline model (such as π₀.₅).
- Cross-scenario generalizationThe model supports fine manipulation (inserting test tubes, picking up screws), manipulation of flexible objects (folding clothes), and control of articulated objects (opening drawers).
The technical principle of lingbot-VA
- Autoregressive diffusion architectureLingBot-VA employs an autoregressive diffusion framework that unifies visual dynamic prediction and action reasoning into a single interleaved sequence, enabling the robot to simultaneously reason about future states and execute precise closed-loop control, achieving a deep fusion of video generation and action decision-making.
- Three-stage processing frameworkThe framework consists of three stages: the autoregressive video generation module predicts future frames based on current observations and language commands; the inverse dynamics model (IDM) decodes specific actions from the predicted video; after the action is executed, the video KV-cache is replaced with real observations, anchoring the model to the actual results and forming closed-loop control.
- Inverse dynamics modelIDM can accurately decode actions from predicted videos and demonstrates good generalization ability across different environments and different robot bodies, serving as a key bridge connecting "imagination" and "execution".
- Real data pre-trainingThe model is pre-trained on a large-scale robot video-motion dataset, learning rich visual dynamics, laying a solid foundation for understanding how the evolution of the physical world works.
The project address for lingbot-VA
- Project official websitehttps://technology.robbyant.com/lingbot-va
- GitHub repositoryhttps://github.com/Robbyant/lingbot-va
- HuggingFace model libraryhttps://huggingface.co/collections/robbyant/lingbot-va
- Technical Papers: https://github.com/Robbyant/lingbot-va/blob/main/LingBot_VA_paper.pdf
Application scenarios of lingbot-VA
- Family Long-Term TasksLingBot-VA can perform complex family tasks that require multi-step planning and long-term memory, such as preparing breakfast and unpacking packages.
- High-precision industrial operationThe model is suitable for precision industrial scenarios that require sub-millimeter level control accuracy, such as inserting test tubes and picking up screws.
- Flexible object processingLingBot-VA can handle deformable objects such as folded clothing, understand material properties, and adapt to dynamic deformation processes.
- Hinged object interactionThe model supports mechanical operations such as opening drawers and accurately handles the relationship between constraints and kinematics.
- Rapid adaptation with few samplesThe model is particularly suitable for data-intensive scenarios where new skills can be learned in just 30-50 demonstrations.