RoboBrain Orca - A multimodal representation world model launched by Zhiyuan. - AiBoss

What is RoboBrain Orca?

RoboBrain Orca, developed by the Beijing Academy of Artificial Intelligence (BAAI), is a multimodal world representation model that replaces traditional next-word/frame/action prediction with next-state prediction, allowing AI to internally construct a unified latent representation of the world. Based on 125,000 hours of video and 160 million event annotations, the model captures natural state transitions through unconscious learning and models semantic causality through conscious learning, condensing world states into a unified latent space. It supports multimodal readouts such as text generation, image prediction, and embodied actions.

The main functions of RoboBrain Orca

World state modelingEncode multimodal world signals (visual, linguistic) into a unified world latent representation space.
Text generationBased on the potential representation of the world, text answers and visual questions are output through the LM head.
Image predictionGiven the current image and instructions, predict the next state image after real-world interaction, rather than simply generating aesthetically pleasing images.
Embodied Action GenerationIt can read robot motion sequences from the world representation, supporting OOD tasks such as dual-arm operation.

The technical principles of RoboBrain Orca

Encoder-Decoder ArchitectureThe Encoder is responsible for learning a unified world latent representation; the Decoder is responsible for reading the representation into specific modal outputs.
Unconscious LearningIt learns dense, natural state transitions from continuous videos, without relying on human labels, by predicting the latent representation of the next frame.
Conscious LearningUsing events described in language and VQA data, we learn sparse but semantically meaningful state transitions to model causality and intent.
Dual-path pre-trainingIt combines the three goals of Observation-only State Transition, Event-conditioned State Transition, and VQA Response Generation.
Freeze the trunk + Lightweight readoutAfter pre-training, the Orca backbone is frozen, and only lightweight modules such as MLP Adaptor, LoRA, or Action Expert are trained downstream.

How to use RoboBrain Orca

Obtain resourcesDownload the open-source code and pre-trained model weight files from the GitHub repository https://github.com/orca-wm.
Prepare dataCollect multimodal world signal data, including continuous video, language event descriptions, and visual question-and-answer pairs.
Access ModuleFreeze the Orca backbone network and connect lightweight readout modules (LM head, image decoder, or Action Expert) according to the target scenario.
Deploy applications: Perform inference in downstream tasks such as robot manipulation, visual question answering, or state prediction, or perform post-training fine-tuning with a small number of in-domain trajectories.

RoboBrain Orca's core advantages

Unified Representation SpaceLanguage, images, and actions are no longer isolated tasks, but different exits from the same potential space of the world.
Physical consistency predictionImage prediction focuses more on real physical processes and state transitions, avoiding the illusions and objects appearing out of thin air that are common in traditional generative models.
Zero-shot generalization abilityIn the action generation task, pre-training did not use any action labels, and training with only 200 in-domain trajectories was able to surpass the dedicated baseline.
Continued Scaling PotentialThe training loss continues to decrease with the amount of data and the size of the model. Currently, only about one-tenth of the data is used, and there is significant room for improvement in the future.

The project address for RoboBrain Orca.

Project official websitehttps://orca-wm.github.io/
GitHub repositoryhttps://github.com/orca-wm/Orca
HuggingFace model libraryhttps://huggingface.co/papers/2606.30534
arXiv technical paper: https://arxiv.org/pdf/2606.30534

RoboBrain Orca's Competitive Comparison

Dimension	Orca	V-JEPA 2.1
mechanism	BAAI (Balance of Artificial Intelligence)	Meta AI
Core Paradigm	Next-State-Prediction (State Transition Modeling)	Next-Frame Prediction (Video Prediction)
Learning path	Unconscious learning + conscious learning (dual path)	Self-supervised video feature learning
Data scale	125,000 hours of video + 160 million event annotations	Large-scale unlabeled videos
Downstream support	Text, image, and action three-modal readout	Primarily supports visual and motor representation
Action generation	With the main branch frozen, OOD generalization can be achieved with just 200 trajectories.	More robot data is needed for fine-tuning.
Understanding state transitions	Explicit modeling of event-level causality and semantic state transition	Emphasis on pixel-level and feature-level consistency

Application scenarios of RoboBrain Orca

Robot operation planningBased on the understanding of world state transitions, it generates physical actions for robotic arms to complete complex interactive tasks such as opening drawers and folding clothes.
Autonomous driving decisionBy predicting the state evolution of vehicles and pedestrians in traffic scenarios, potential risks can be anticipated in advance and safe driving routes can be planned.
Embodied agent trainingAs a universal world representation interface, it provides pre-trained knowledge for different robot platforms, reducing the data acquisition cost for new tasks.
Interactive Video UnderstandingIt predicts the subsequent development of videos based on natural language instructions, and supports causal reasoning, question answering, and event development deduction.
Physical simulation and digital twinTo construct reasonable and predictable world state models for industrial virtual environment simulation and physical process simulation.