VLAC - A large-scale embodied reward model open-sourced by the Shanghai AI Lab

What is VLAC?

VLAC is an embodied reward model released by the Shanghai Artificial Intelligence Laboratory. Based on the InternVL multimodal model, it integrates internet video data and robot operation data to provide process rewards and task completion estimates for robot reinforcement learning in the real world. VLAC can effectively distinguish between normal progress and abnormal/stagnant behavior, and supports rapid generalization with few samples through in-context learning. It has local smoothing and negative reward mechanisms to ensure the stability and effectiveness of reinforcement learning. VLAC not only outputs reward signals but also outputs robot action commands, helping robots to learn autonomously and adapt quickly to new scenarios in the real world. VLAC supports human-robot collaboration mode to further improve training efficiency.

Main functions of VLAC

• Provide process rewards and completion estimatesIt provides continuous and reliable supervision signals for robot reinforcement learning in the real world, determines whether the task has been completed, and estimates the progress of completion.
• Distinguishing between normal and abnormal behaviorEffectively identify normal propulsion, abnormal or stalled behavior during robot operation to avoid ineffective exploration.
• Supports rapid generalization with small sample sizesBy using in-context learning, we can achieve rapid generalization from small samples and improve the model's adaptability in new scenarios.
• Output robot motion commandsWhile providing reward signals, it can also output action instructions for the robot to execute, helping the robot to learn and adjust its behavior autonomously.
• Building a reinforcement learning frameworkThe VLA reinforcement learning framework built around VLAC enables robots to quickly adapt to new scenarios in real-world interactions, improving task success rates.
• Supports human-computer collaboration modeBy employing various human-computer collaboration paradigms, we can further enhance training flexibility and improve learning efficiency.

VLAC Technical Principles

• Multimodal fusionBased on the InternVL multimodal large model, it integrates data from multiple modalities such as vision and language to improve the comprehensive understanding of tasks and environments.
• Data-driven reward generationBy leveraging internet video data and robot operation data, dense reward signals are generated through learning, providing stable feedback for reinforcement learning.
• Task progress estimationBy leveraging the model's real-time understanding of the task, the progress of task completion can be estimated, providing process rewards for reinforcement learning.
• Abnormal behavior detectionBy analyzing robot operation data, abnormal or stagnant behaviors can be identified, ineffective exploration can be avoided, and learning efficiency can be improved.
• Context learning mechanismIt supports in-context learning, enabling rapid adaptation to new tasks with a small number of samples, thereby improving the model's generalization ability.
• Action instruction generationWhile providing reward signals, it generates robot action commands to achieve closed-loop control from perception to action.
• reinforcement learning framework integrationThe goal is to build a VLA reinforcement learning framework that combines process rewards and task completion to improve the robot's learning and adaptation capabilities in the real world.
• Human-machine collaboration enhancementThe training process of the model can be further optimized through human-computer collaboration, such as expert data playback and manual assistance exploration.

VLAC's project address

• Project official websitehttps://vlac.intern-ai.org.cn
• Github repositoryhttps://github.com/InternRobotics/VLAC
• HuggingFace model libraryhttps://huggingface.co/InternRobotics/VLAC

VLAC Application Scenarios

• Robot reinforcement learningIt provides process rewards and task completion estimates for robots in real-world reinforcement learning, helping robots to quickly adapt to new tasks and environments.
• Human-machine collaborative tasksIt supports human-machine collaboration mode, and improves the flexibility and efficiency of robot training through expert data playback, manual assistance and exploration, etc.
• Multi-robot collaborative learningIn multi-robot environments, the VLA reinforcement learning framework enables multiple robots to interact and learn simultaneously in the real world, improving task success rates.
• Complex Task Decomposition and LearningThe complex task is broken down into multiple sub-tasks, and a reward signal is provided for each sub-task to help the robot complete the complex task step by step.
• Quickly adapt to new scenariosBy leveraging its ability to rapidly generalize from small samples, the robot can quickly learn and adapt to new scenarios, thereby improving task completion rates.