Embodied Reasoner - An embodied interactive reasoning model developed by Zhejiang University in collaboration with Alibaba and other institutions.

What is Embodied Reasoner?

Embodied Reasoner is a novel embodied interactive reasoning model developed by Zhejiang University, the Institute of Software, Chinese Academy of Sciences, Alibaba Group, and other institutions. It leverages visual search, reasoning, and action collaboration to complete complex tasks. The model employs a three-stage training method—imitation learning, self-exploration, and self-correction—to generate diverse thought processes (such as contextual analysis, spatial reasoning, and self-reflection), enabling efficient planning and reasoning based on interaction history and spatial layout. In various tasks on the AI2-THOR simulator, Embodied Reasoner significantly outperforms existing visual reasoning models, demonstrating excellent performance in complex, long-term tasks and reducing repetitive searches and logical inconsistencies.

Main functions of Embodied Reasoner

• Visual search and target localizationSearch for hidden or exposed objects in complex environments and locate targets according to mission requirements.
• Reasoning and PlanningBased on generating diverse thinking processes (such as situational analysis, spatial reasoning, self-reflection, etc.), formulate efficient action strategies.
• Operation executionBased on the reasoning results, perform corresponding actions, such as navigation, grabbing, and placing, to complete the task.
• Self-correction and learningBased on reflection and self-correction mechanisms, it avoids duplicate searches and logical inconsistencies, thereby improving task success rates.
• Complex task processingProficient in handling complex tasks with long time sequences and multiple steps.

The technical principles of Embodied Reasoner

• Data EngineIt automatically generates task instructions and corresponding "observation-thinking-action" trajectories based on task templates and scene metadata, including rich thought processes and interactive images.
• Three-stage training:
  • Imitation learningFine-tune the synthesized trajectory and learn basic interaction skills.
  • Self-exploration (refusal to sample)Based on the trajectories generated by sampling and evaluation, the model's exploratory capabilities are enhanced.
  • Self-correction (reflection and adjustment)Injecting abnormal states and reflecting on and correcting errors enhances the model's adaptability.
• Multimodal interactionIt combines visual input (images) and language output (thinking and actions) to achieve efficient environmental interaction and task completion.
• Reasoning mechanismBased on generating long thought sequences, it simulates the human reasoning process and improves the model's performance in complex tasks.

Embodied Reasoner's project address

• Project official website:https://embodied-reasoner.github.io/
• GitHub repository:https://github.com/zwq2018/embodied_reasoner
• HuggingFace model library:https://huggingface.co/datasets/zwq2018/embodied_reasoner
• arXiv technical paper:https://arxiv.org/pdf/2503.21696

Application scenarios of Embodied Reasoner

• Smart HomeIt helps users find items and operate appliances at home.
• Warehousing and LogisticsAutomatically locate and move goods in the warehouse, optimizing warehouse management.
• Medical assistance: Assist medical staff in finding and organizing items in hospitals or nursing homes.
• Industrial AutomationPerforming complex operational tasks in the factory, such as parts handling and equipment maintenance.
• Education and ResearchIt can be used as an educational tool to help students understand task planning, or in research on human-computer interaction and robot intelligence.