VPP - The first large-scale AIGC robot model launched by Tsinghua University and Xingdong Era
VPP (Video Prediction Policy) is the first large-scale AIGC robot model launched by Tsinghua University and Xingdong Era. Based on a pre-trained video diffusion model, it learns from a large amount of video data on the Internet to directly predict future scenes...
What is VPP?
VPP (Video Prediction Policy) is the first AIGC (AI-Generated Collective) robot model launched by Tsinghua University and Star Era. Based on a pre-trained video diffusion model, it learns from a large amount of video data on the internet to directly predict future scenes and generate robot actions. VPP can predict the future in advance, achieve high-frequency prediction and action execution, support switching between humanoid robot bodies, and significantly reduce the dependence on high-quality real robot data. VPP achieved a near-perfect score in the Calvin ABC-D benchmark test and performed excellently in complex and dexterous maneuvering tasks in the real world. VPP's open-source nature provides strong technical support for the development of embodied intelligent robots.
Main functions of VPP
- Predicting future scenariosThis allows robots to "see" the future before taking action, enhancing their generalization capabilities.
- High-frequency prediction and action executionIt achieves a prediction frequency of 6-10Hz and a control frequency of over 50Hz, improving the smoothness of motion.
- Cross-robot ontology learning: Directly learn video data of robots of different forms, including human operation data, to reduce data acquisition costs.
- Multi-task learning and generalizationIt excels in complex real-world tasks such as grabbing, placing, stacking, pouring water, and using tools.
- Explainability and Debugging OptimizationBased on predictive video, failure scenarios can be identified in advance, making it easier for developers to make targeted optimizations.
VPP technical principle
- Predictive visual representation of video diffusion model (VDM)This involves learning to predict future scenes based on pre-trained video diffusion models (such as Stable Video Diffusion). The video diffusion model generates predictive visual representations based on single-step denoising, which include the current frame and can explicitly represent future frames.
- motor learningVideo Former is used to aggregate predictive visual representations and extract spatiotemporal information. Robotic actions are generated based on a diffusion policy, achieving a seamless transition from prediction to execution.
- Optimization and GeneralizationVPP is trained using internet video data and robot operation data, reducing its reliance on high-quality real-machine data. Based on cross-entity learning, VPP can directly learn from video data of robots with different forms, improving the model's generalization ability.
VPP project address
- Project official website:https://video-prediction-policy.github.io/
- GitHub repository:https://github.com/roboterax/video-prediction-policy
- arXiv technical paper:https://arxiv.org/pdf/2412.14803
VPP application scenarios
- Home services: Perform household chores (pour water, fetch things), take care of the elderly or children (deliver items).
- Industrial manufacturingUsed for parts picking, cargo handling and stacking to improve production efficiency.
- Medical assistance: Assist with the delivery of surgical instruments, rehabilitation training, and ward supplies.
- Education and ResearchIt helps students understand complex operational procedures and apply them in laboratory experiments.
- service industryServices include restaurant delivery, hotel luggage handling, and guided tours in public places.