MobA - A mobile intelligent agent developed by Shanghai Jiao Tong University
MobA (Mobile Agent) is a novel mobile intelligent agent developed by a team at Shanghai Jiao Tong University. It enhances the automated task execution capabilities of mobile devices based on multimodal large language models (MLLMs). MobA employs a two-tier architecture: a high-level...
What is MobA?
MobA (Mobile Agent) is a novel mobile intelligent agent developed by a team at Shanghai Jiao Tong University. It enhances the automated task execution capabilities of mobile devices based on Multimodal Large Language Models (MLLMs). MobA employs a two-tier architecture: a high-level global agent (GA) responsible for understanding user commands, managing historical records, and planning tasks; and a low-level local agent (LA) that executes specific actions according to the GA's plans. The system's built-in reflection module enables MobA to efficiently handle complex tasks, including those previously unseen.
MobA's main functions
- Understanding User CommandsIt understands the user's natural language commands and translates them into executable tasks.
- Task planning and decompositionThe advanced global agent (GA) is responsible for breaking down complex tasks into smaller, more manageable subtasks.
- Action executionThe low-level local agent (LA) executes specific sub-tasks and actions according to the guidance of the GA.
- Historical Memory TrackingGA tracks historical memories and considers past experiences and information when planning new tasks.
- Reflection and Self-OptimizationThe integrated reflection module allows MobA to perform self-evaluation after executing a task, optimizing future task execution.
- Cross-application operationHandles complex tasks involving multiple applications, enabling automated operations across applications.
MobA's technical principles
- Multimodal Large Language Models (MLLMs)Based on MLLMs, the model can process and understand various types of data, such as text and images.
- Two-level intelligent agent architectureThe system consists of a high-level global agent (GA) and a low-level local agent (LA), which are responsible for task planning and action execution, respectively.
- Task planning moduleThe planning module in GA is responsible for breaking down user instructions into a series of subtasks and evaluating the feasibility of the tasks.
- Action ModuleThe action module in LA is responsible for identifying whether a task can be completed in one step and providing the corresponding action instructions.
- Memory moduleThe system includes a memory module used to store and update task execution traces, user preferences, and application information.
MobA's project address
- arXiv technical paper:https://arxiv.org/pdf/2410.13757
MobA Application Scenarios
- Personal AssistantIn daily life, it acts as a personal assistant, helping users manage schedules, set reminders, and query information.
- Smart Home ControlIt integrates with smart home devices, allowing you to control smart devices in your home, such as lights, temperature control, and security monitoring, based on voice or text commands.
- Mobile device automationOn mobile devices, it automates complex workflows such as filling out forms, managing emails, and optimizing application settings.
- Assistance for the elderly and disabledTo provide assistance to the elderly and people with disabilities, simplify the process of using smart devices, and improve their quality of life.
- Education and LearningIn the field of education, it helps students access information, manage learning materials, and even assist in completing assignments.