Gen2Act - Google, Carnegie Mellon, and Stanford jointly launch a strategy for generating human-like videos to guide robot operation.
Gen2Act is a robot manipulation strategy jointly developed by Google, Carnegie Mellon University, and Stanford University. It generates human-like videos based on motion information from predictive network data and uses these videos to guide robots in performing new tasks...
What is Gen2Act?
Gen2Act, a robot manipulation strategy jointly developed by Google, Carnegie Mellon University, and Stanford University, generates human-like videos based on motion information from predictive network data and uses these videos to guide robots in performing new tasks. The strategy leverages a large amount of available online video data, avoiding the complexity of directly generating robot videos. The core of Gen2Act lies in zero-shot human-like video generation, combining a pre-trained video generation model with a small amount of robot interaction data to train the strategy. In real-world applications, Gen2Act demonstrates strong generalization capabilities, able to manipulate unfamiliar object types and perform new actions, with a significantly improved success rate compared to other methods. Gen2Act supports long-term execution of complex tasks, such as continuously performing multi-step activities like "making coffee." This method reduces the need for large-scale robot data acquisition and uses a closed-loop strategy to dynamically adjust and improve the accuracy of the operation.
Main functions of Gen2Act
- Zero-shot video generationGen2Act can directly use pre-trained video generation models to generate videos of humans performing tasks based on language-described tasks and scene images, without the need for task-specific fine-tuning.
- Generalization to new tasksBased on generated human video guidance, Gen2Act enables robots to perform new tasks not seen in the training data, including manipulating unfamiliar object types and performing new actions.
- Closed-loop strategy executionBy combining the generated video and the robot's real-time observation, Gen2Act dynamically adjusts the robot's actions based on a closed-loop strategy to accurately execute tasks in response to changes in the scene.
- Long-duration task processingGen2Act can complete a single task, or, based on the linking of task sequences, execute a series of complex, long-running tasks, such as "making coffee," which involves multiple consecutive steps.
- Reduce data requirementsGen2Act requires less robot demonstration data, greatly reducing the cost and workload of data collection.
The technical principles of Gen2Act
- Human video generation: Based on a pre-trained video generation model, zero-shot videos of humans performing tasks are generated from the first frame image of the task and scene described in language.
- Video to motion translation: Based on a closed-loop strategy, generated human videos are transformed into robot actions. The strategy uses visual features and point trajectories from the video to predict implicitly encoded motion information.
- Visual feature extraction: Features were extracted from the generated video and the robot's observation history using the ViT encoder and Transformer encoder.
- Point trajectory prediction: The Transformer is used to predict the motion trajectory of points in a video, and an auxiliary loss training strategy is employed.
- Behavioral cloning loss: It uses an optimization strategy that minimizes the error between predicted and actual actions to mimic human behavior in videos.
Gen2Act's project address
- Project official websitehttps://homangab.github.io/gen2act/
- arXiv technical paper:https://arxiv.org/pdf/2409.16283
Application scenarios of Gen2Act
- Home AutomationIn a home environment, Gen2Act can control home appliances such as turning on and off the microwave, operating the coffee machine, and organizing items, helping to achieve home automation.
- Industrial AutomationIn manufacturing, Gen2Act can perform complex assembly tasks or replace or assist manual operations in environments that require flexibility and adaptability.
- service industryIn the catering or retail service industry, Gen2Act guides robots to complete tasks such as taking orders, serving food, and organizing shelves.
- Medical assistanceIn the medical field, Gen2Act helps develop robots that perform delicate operations, such as assisting in surgery or delivering medical supplies.
- Disaster reliefAt disaster relief sites, Gen2Act controls robots to carry out search and rescue missions in unknown environments.