Mobile-Agent - Alibaba's open-source, self-developed multimodal mobile device intelligent agent.

What is Mobile-Agent?

Mobile-Agent is a mobile, intelligent agent software entity that can move between nodes in a network to perform tasks on behalf of users or other agents. It can interrupt its current execution, move to another device to resume operation, and promptly return results. This mobility aims to bring program execution as close to the data source as possible, reducing network communication overhead, saving bandwidth, balancing load, accelerating task execution, and improving the processing efficiency of distributed systems.

Mobile-Agent applications are rapidly developing, particularly in the field of mobile device operation. For example, Mobile-Agent-v2, jointly proposed by Alibaba and Beijing Jiaotong University, is a mobile device operation assistant that achieves effective navigation through multi-agent collaboration. Based on a multimodal large language model (MLLM), it can autonomously complete complex mobile device operation tasks. Mobile-Agent-v2 includes three professional roles: a planning agent, a decision-making agent, and a reflection agent. It can generate tasks based on historical operations and perform self-reflection and adjustments during the operation process. Mobile-Agent also supports purely visual solutions, eliminating the need for system UI files and understanding and operating the phone by analyzing images. This enables flexible application in different mobile operating system environments.

The third-generation GUI intelligent agent framework, Mobile-Agent-v3, is based on the visual multimodal model GUI-Owl and achieves a major breakthrough in GUI automation technology. It covers PC, Web, and mobile systems and can accurately identify interface elements and perform operations on multiple platforms.

Main functions of Mobile-Agent

• Operation positioningMobile-Agent can accurately identify and tap specific icons and text on the screen. It determines the location of the action through detection models and visual perception tools, such as using OCR tools to locate text or icon detection tools to identify icons.
• Self-planningBased on user instructions and the current screen state, Mobile-Agent can automatically plan and execute a series of steps until the task is completed. It iteratively acquires screenshots, combining them with operation history and system prompts to determine the next step.
• Self-reflectionDuring operation, if errors or invalid operations occur, Mobile-Agent can promptly detect them and take remedial measures. For example, if the screen does not change or displays an error page after an operation, it will try other operations or adjust parameters.
• Multi-application operationMobile-Agent supports automated operations across multiple applications, such as switching between different applications and collaborating to complete tasks.
• Pure vision solutionsIt does not rely on system XML files or other metadata; it understands and operates the phone by analyzing images, thus making its operation unrestricted.

The technical principle of Mobile-Agent

• Multimodal large language modelMobile-Agent combines a large-scale language model (such as GPT-4V) to understand and execute the user's natural language commands. The model can generate corresponding operation steps by reading screenshots and user commands.
• Visual perception technology
  • Text and icon detectionThe system uses Optical Character Recognition (OCR) tools to locate text on the screen and icon detection tools and CLIP models to identify the precise location of icons. This enables the Mobile-Agent to accurately identify and locate elements on the screen and perform precise operations.
  • Screenshot AnalysisMobile-Agent obtains the visual information needed for operation by analyzing the current screenshot. Using the screenshot as input, combined with operation history and user commands, it determines the next step.
• Multi-agent cooperation mechanismMobile-Agent employs a multi-agent architecture, comprising multiple specialized agents such as visual perception agents, decision-making agents, execution agents, and reflective agents. These agents each perform their specific functions and collaborate to complete complex mobile device operation tasks.
• Autonomous mission planning and execution
  • Self-planningMobile-Agent can automatically plan and execute a series of operation steps based on user instructions and the current screen state. It iteratively captures screenshots, processes prompts and operation history, and generates the next step.
  • Self-reflectionDuring operation, if errors or invalid actions occur, Mobile-Agent can promptly detect them and take remedial measures. For example, if the screen does not change or displays an error page after an operation, it will try other operations or adjust parameters.
• Prompt formatMobile-Agent uses the prompt format from ReAct, requiring the agent to output three parts: Observation, Thought, and Action. This helps the agent better understand and execute tasks.

Mobile-Agent project address

• Github repositoryhttps://github.com/X-PLUG/MobileAgent
• arXiv technical paper: https://arxiv.org/pdf/2401.16158
• Experience the demo onlinehttps://huggingface.co/spaces/junyangwang0410/Mobile-Agent

Application scenarios of Mobile-Agent

• Mobile device operation assistantMobile-Agent can perform various operations on mobile devices based on user instructions, such as opening applications, clicking buttons, and entering text, thus improving ease of use.
• Multi-application collaborative operationIt supports switching and collaborative task completion between different applications, such as copying information from one application and pasting it into another.
• Automated testingUsed to simulate user actions and perform automated testing of mobile applications, improving testing efficiency and accuracy.
• Intelligent navigation and task planningAutomatically plan and execute a series of operations based on user goals and current status until the task is completed, such as finding and executing specific functions in complex application environments.
• Visual perception-driven operationThrough a pure vision solution, it analyzes only images to understand and operate mobile phones, and can be flexibly applied in different operating systems without being limited by system data structures.