RoboBrain - An open-source embodied brain model from the Academy of Artificial Intelligence.

What is RoboBrain?

RoboBrain is an open-source embodied brain model launched by the Beijing Academy of Artificial Intelligence (BAAI), propelling single-machine intelligence towards swarm intelligence. It consists of three modules: a base model for task planning, an A-LoRA module for manipulable area perception, and a T-LoRA module for trajectory prediction. RoboBrain employs a multi-stage training strategy, possesses long historical frame memory and high-resolution image perception capabilities, and can map abstract instructions to concrete actions. It has demonstrated excellent performance in evaluation tasks such as task planning, manipulable area perception, and trajectory prediction.

RoboBrain's main functions

• Planning CapabilityBreak down complex operation instructions into manageable subtasks. For example, breaking down "lift the teapot and pour water into the cup" into steps such as "approach the teapot and lift it", "move the teapot so that the spout is aligned with the rim of the cup", and "tilt the teapot to pour the water".
• Affordability Perception: Identify and interpret the operable areas of interactive objects, such as the handle or spout of a teapot.
• Trajectory PredictionPredict the complete trajectory required to complete the operation, such as the movement trajectory from the current position to the teapot handle.

RoboBrain's technical principles

• Model ArchitectureRoboBrain is based on the LLaVA framework and consists of the following three main modules:
  • Visual EncoderThe SigLIP model is used to encode the input image into visual features.
  • ProjectorVisual features are mapped to the same dimension as text embeddings using a two-layer MLP.
  • Large Language Model (LLM)It adopts the Qwen2.5-7B-Instruct model to understand and generate text instructions.
• Multi-stage training strategyRoboBrain employs a multi-stage training strategy to improve its performance in robotic manipulation tasks.
  • OneVision Training: Pre-train on large-scale general vision datasets to develop basic visual and language understanding capabilities.
  • Robot Task TrainingFine-tuning on the ShareRobot dataset enhances task planning, operational area awareness, and trajectory prediction capabilities.
• Dataset supportRoboBrain's training relies on the ShareRobot dataset, a high-quality heterogeneous dataset containing multi-dimensional annotations of task planning, manipulable object regions, and end-effector trajector trajector dimensionality. The dataset's diversity and accuracy are carefully designed to support the model's performance on complex tasks.
• Reasoning processIn practical applications, RoboBrain first senses visual input, decomposes the input instructions into a series of executable subtasks, and then performs operable area perception and trajectory prediction. This step-by-step processing approach enables the model to efficiently translate abstract instructions into concrete robot operations.

RoboBrain's project address

• Project official website:https://superrobobrain.github.io/
• Github repository:https://github.com/FlagOpen/RoboBrain
• HuggingFace model library:https://huggingface.co/BAAI/RoboBrain
• arXiv technical paper:https://arxiv.org/pdf/2502.21257

Application scenarios of RoboBrain

• Multi-robot collaborationRoboBrain, as the core brain model of RoboOS, a cross-ontology embodied cerebellum collaboration framework, enables efficient collaboration between multiple different types of robots.
• Complex task planningRoboBrain can break down complex operation instructions into manageable subtasks, such as "Water plants", "Put the pot in the drawer", and "Cluster blocks of the same color into different corners". RoboBrain can generate detailed planning steps.
• Operable area perceptionRoboBrain can identify and interpret the operable areas of interactive objects. For example, in the task of "Cluster blocks of the same color into different corners", RoboBrain can identify the operable areas of blocks of different colors and plan a reasonable operation path.
• Real-time feedback and optimizationRoboBrain, combined with RoboOS's edge-cloud collaboration capabilities, can receive execution feedback in real time, dynamically adjust strategies according to environmental changes, continuously optimize task planning, and improve robustness.