AB
AB
AiBoss

All Categories

project

Gemini Robotics - A large-scale embodied intelligence model launched by Google DeepMind

Gemini Robotics is a robotics project launched by Google DeepMind, based on Gemini 2.0, which brings the capabilities of large-scale multimodal models to the physical world. The project includes two main models: Gemini Robotics-ER and...

What is Gemini Robotics?

Gemini Robotics is a robotics project launched by Google DeepMind based on Gemini 2.0, bringing the capabilities of large multimodal models to the physical world. The project comprises two main models: Gemini Robotics-ER and Gemini Robotics. Gemini Robotics-ER is an enhanced Visual-Language Model (VLM) with powerful embodied reasoning capabilities, supporting tasks such as understanding 3D space, object detection, pointing, trajectory prediction, and grasping prediction. Gemini Robotics is a Visual-Language-Motion (VLA) model that supports direct control of robots to complete complex maneuvers, exhibiting robustness to changes in object type and position, and executing open-vocabulary instructions. With further fine-tuning, Gemini Robotics can solve long-term, highly flexible tasks (such as origami or card playing) and quickly adapt to new tasks and robot forms.

Gemini Robotics' main functions

  • Execution of complex tasks in intelligent dialogue and question answering:
    • Highly flexible tasksExamples include origami, playing cards, and using tools.
    • Long-term taskTasks involving multiple steps, such as packing lunch boxes or making sandwiches.
    • Precise operationExamples include using tweezers to pick up small objects or tighten bottle caps.
  • Strong visual and language comprehension abilities:
    • Visual understandingIt can identify and locate objects in a scene, supporting multi-view understanding, 3D spatial reasoning, and object detection.
    • Language comprehension: Understand natural language instructions and perform open vocabulary tasks.
  • Environmental adaptability and generalization ability:
    • Visual generalizationIt is robust to changes in background, lighting, and interfering objects in the scene.
    • Action generalization: Operations that adapt to changes in object position and different object instances.
    • Instruction generalization: Understand different ways of expressing instructions (such as different languages, instructions with spelling errors).

Gemini Robotics' technical principles

  • Gemini 2.0 Basic ModelBased on Gemini 2.0 extensions. Gemini 2.0 provides Gemini Robotics with powerful vision and language understanding capabilities.
  • Embodied reasoningGemini Robotics-ER is an extended version of Gemini Robotics, focusing on embodied reasoning capabilities. It supports understanding three-dimensional space, object relationships, and motion trajectories in the physical world.
  • Visual-Language-Motion (VLA) Model:
    • Visual inputIt uses a camera to capture scene images and identify objects and the environment.
    • Language instructionsIt understands natural language instructions and translates them into specific robot actions.
    • Action outputGenerates motion commands for the robot to perform complex tasks.
  • Data-driven training:
    • Robot motion dataBased on real robot operation data, it learns how to perform various tasks.
    • Non-robot dataIt provides rich background knowledge and generalization capabilities for online images, text, videos, etc.
  • Architecture Design:
    • Cloud-based VLA backbone network: Responsible for handling complex visual and linguistic reasoning tasks.
    • Local Action DecoderRunning locally on the robot, it is responsible for generating motion commands in real time, ensuring low latency and high responsiveness.

Gemini Robotics' project address

Application scenarios of Gemini Robotics

  • Industrial manufacturingUsed in complex assembly, quality inspection and repair to improve production efficiency and accuracy.
  • Logistics warehousingTo automate cargo sorting, packaging, and loading/unloading, thereby improving logistics efficiency.
  • Home servicesAssist with household chores, such as tidying, cleaning, and caring for the elderly or disabled.
  • HealthcareIt assists in rehabilitation training and surgical procedures, and supports medical processes.
  • Educational ResearchIt serves as an educational tool and research assistant, supporting teaching and experimental operations.
Back