Optimus-1 - An intelligent agent framework jointly developed by Harbin Institute of Technology and Pengcheng Laboratory
Optimus-1 is an intelligent agent framework developed by Harbin Institute of Technology (Shenzhen) and Pengcheng Laboratory, designed to address the challenges of completing long-term tasks in open-world environments. The framework combines structured knowledge and multimodal experience, enabling intelligent agents to...
What is Optimus-1?
Optimus-1 is an intelligent agent framework developed by Harbin Institute of Technology (Shenzhen) and Pengcheng Laboratory, designed to address the challenges of completing long-term tasks in open-world environments. The framework combines structured knowledge and multimodal experience, enabling agents to better perform complex tasks. Optimus-1 includes a hybrid multimodal memory module composed of a hierarchical directed knowledge graph (HDKG) and an abstract multimodal experience pool (AMEP), along with a knowledge-guided planner and an experience-driven reflector, significantly improving long-term task performance in environments such as Minecraft, approaching human-level performance.
Main functions of Optimus-1
- Knowledge-Guided PlannerIt generates executable sub-objective sequences based on structured knowledge (HDKG) to plan and execute long-term tasks.
- Experience-Driven ReflectorAssess the current mission status based on historical experience (AMEP) and adjust the action plan as necessary.
- Action Controller: Execute sub-goals generated by the planner and interact with the environment to complete the task.
- Hybrid Multimodal MemoryCombining HDKG and AMEP, it stores and utilizes world knowledge and historical experience to support complex decision-making and adaptive learning.
- Self-EvolutionBased on the "free exploration-teacher guidance" method, it can gradually improve memory capacity and task execution ability without parameter updates.
Technical principles of Optimus-1
- Hierarchical Directed Knowledge Graph (HDKG):
- Transform world knowledge (such as item crafting rules) into a graph structure, where nodes represent objects and edges represent relationships between objects (such as crafting relationships).
- Based on graph retrieval and topological sorting, it provides agents with the knowledge of the materials and tools needed to complete tasks.
- Abstract Multimodal Experience Pool (AMEP):
- Dynamically summarize and store multimodal information (such as environment, status, plan, visual frames, etc.) during task execution.
- The MineCLIP model is used to calculate the similarity between visual objects and sub-objects, and to store highly relevant visual memories to provide a reference for reflection.
- Knowledge-guided plannerBy combining visual observation and knowledge from HDKG, a reasonable sequence of sub-targets is generated to guide the actions of the action controller.
- Experience-driven reflectorPeriodically activate the system to retrieve multimodal experiences related to the current sub-goal from AMEP, assess the current task status, and decide whether to continue execution, complete the task, or replan it.
- Action controllerBased on the sub-goal and the current observation, generate low-level actions required to interact with the environment, such as mouse and keyboard operations.
Project address of Optimus-1
- Project official website:cybertronagent.github.io/Optimus-1
- GitHub repository:https://github.com/JiuTian-VL/Optimus-1
- arXiv technical paper:https://arxiv.org/pdf/2408.03615
Application scenarios of Optimus-1
- Game environment automationIn complex game environments such as Minecraft, Optimus-1 automates tasks such as resource gathering, building construction, and exploration, providing an immersive gaming experience.
- Virtual personal assistantIn virtual reality or augmented reality applications, it can serve as a personal assistant to users, helping to manage schedules, provide information retrieval services, and automate daily tasks.
- Home robot assistantIn smart home systems, this involves guiding home service robots to perform long-term tasks such as cleaning, security monitoring, and routine maintenance, thereby improving the level of home automation.
- Industrial Automation and OptimizationIn the industrial manufacturing sector, optimizing production processes and guiding robots to automatically complete complex assembly line tasks improves production efficiency and safety.
- Simulation training and educationIn military training or medical education, create complex simulation environments to help trainees learn decision-making and operational skills through simulated practice.