Gemini Robotics On-Device - Google's first locally embodied intelligent model

What is Gemini Robotics On-Device?

Gemini Robotics On-Device is Google DeepMind's first visual-language-motion (VLA) model that can run locally on a robot. The model boasts powerful offline operational capabilities, able to perform fine-grained tasks following natural language commands, such as opening bags and folding clothes. It supports deployment on various robot bodies, exhibits low latency, and is suitable for latency-sensitive applications. Gemini Robotics On-Device quickly adapts to new tasks, learning new actions with only 50 to 100 demo samples, demonstrating strong generalization performance. Google has released the Gemini Robotics SDK to help developers evaluate and deploy the model, reducing development costs and risks.

Main functions of Gemini Robotics On-Device

• Local offline operationGemini Robotics On-Device runs entirely locally on the robot, eliminating the need for cloud computing and resolving issues of network latency and unstable connections. This allows the robot to perform tasks stably even in environments with no network connection or weak network signal.
• Follow natural language instructionsThe model can understand human natural language commands. It can handle complex, multi-step instructions, allowing the robot to truly operate according to human intentions.
• Complete the task of fine operationIt supports a variety of robot bodies, from humanoid robots to industrial dual-arm robots, and can complete various tasks that require fine manipulation, such as opening bags, folding clothes, zipping lunch boxes, pulling cards, pouring salad dressing, and assembling industrial-grade belts.
• Quickly adapt to new tasksGoogle has opened up fine-tuning capabilities for its VLA model for the first time, allowing developers to adapt the model to entirely new tasks with just 50 to 100 demo samples. Even for the most complex tasks, a fairly high success rate can be achieved with fewer than 100 samples.
• Cross-platform deploymentThe model can be transferred to completely different robotic platforms, such as the dual-arm Franka FR3 robot and Apptronik's Apollo humanoid robot, demonstrating strong generalization ability.

Gemini Robotics On-Device Technology Principles

• Multimodal reasoning abilityGemini Robotics On-Device leverages the multimodal reasoning capabilities of Gemini 2.0 to simultaneously process information from multiple modalities, including vision, language, and motion. It perceives the environment based on visual input, understands language commands to determine task objectives, and generates corresponding actions to complete the task.
• Optimized model architectureTo enable local operation, the model has been optimized to reduce computational resource requirements while maintaining strong performance. The model can perform low-latency inference on robotic devices, ensuring real-time task execution.
• Fine-tuning functionAs Google's first VLA model that can be fine-tuned, developers can fine-tune the model based on a small number of demo samples, allowing it to adapt to new tasks and environments. This fine-tuning feature enables the model to quickly learn new skills, improving the robot's adaptability and flexibility.
• Security MechanismThe model is based on a holistic security solution that emphasizes both semantic and physical safety. It leverages the Live API to capture semantic and content safety issues, preventing the robot from performing potentially dangerous or inappropriate actions. It interfaces with the underlying safety-critical controller to ensure the robot's actions comply with physical safety requirements, guaranteeing the robot's safety during task execution.

Gemini Robotics On-Device project address

• Project official website: https://deepmind.google/discover/blog/gemini-robotics-on-device-brings-ai-to-local-robotic-devices/

Gemini Robotics On-Device Application Scenarios

• Industrial manufacturingOn industrial production lines, it performs complex assembly tasks, such as assembling automotive parts and precision installing electronic equipment, thereby improving production efficiency and quality.
• Logistics warehousingIt assists in handling goods and managing inventory, identifies goods information, classifies and stacks them according to instructions, optimizes logistics processes, and reduces human error.
• Medical careIt assists medical staff in passing surgical instruments and providing guidance on rehabilitation training, providing precise care for patients and reducing the workload of medical staff.
• Home servicesIt helps with household chores, such as cleaning, tidying up, and caring for the elderly and children, improving the convenience and comfort of life.
• Retail servicesIn shopping malls, supermarkets and other places, we provide customers with services such as product information inquiry, shopping guidance and goods handling to enhance the shopping experience.