AppAgentX - A self-evolving GUI proxy framework developed by Westlake University.

What is AppAgentX?

AppAgentX, developed by Westlake University, is a novel self-evolving GUI (Graphical User Interface) agent framework. It enhances the efficiency and intelligence of agents in smartphone interactions by abstracting high-level actions from execution history. AppAgentX combines memory and evolutionary mechanisms to record task execution processes, identify repetitive operation sequences, and replace them with more efficient high-level actions, reducing reliance on hierarchical reasoning. Based on a chain-like knowledge framework, AppAgentX continuously optimizes behavior, enhancing adaptability and efficiency. AppAgentX significantly outperforms existing methods in multiple benchmark tests, demonstrating higher accuracy and efficiency, providing new insights for the application of intelligent agents in complex task execution.

Main functions of AppAgentX

• Automatically summarize efficient operation modesIt detects repetitive operations during task execution and automatically summarizes them into high-level "one-click" operations, simplifying the operation process.
• Reduce redundant calculations and improve execution efficiencyBased on memory and reuse execution strategies, it avoids repetitive reasoning and makes task execution more efficient.
• Vision-based general operational capabilitiesIt relies on screen visual information for operation, requires no backend API support, is universal across different software and devices, and achieves "plug and play".
• Supports complex tasks and cross-application operationsIt can operate various applications like a human, supporting complex cross-application tasks, such as crawling information from web pages and filling it into Excel, or operating in conjunction with multiple software programs.

AppAgentX Technical Principles

• Memory mechanismThis system records task execution history using a chain-like structure, including page nodes and element nodes. Page nodes record UI page descriptions and element information, while element nodes record specific interaction details. LLM is used to generate functional descriptions for pages and elements, merging duplicate descriptions to form a unified record.
• Evolutionary mechanismAnalyze task execution history to identify recurring low-level operation sequences. Abstract these recurring sequences into high-level actions (called "shortcut nodes"), replacing the original low-level operations. Expand the action space by incorporating high-level actions into the agent's operation set, improving execution efficiency.
• Chain knowledge frameworkUse a graph-based storage structure (such as Neo4j) to record the nodes and relationships of task execution. Nodes include pages, elements, and high-level actions, and relationships represent the interactions and transformations between them. Use a chained structure to achieve continuous optimization and evolution of behavior.
• Task execution processDuring task execution, the agent identifies the current page and elements based on visual matching. If a high-level action is matched, the corresponding low-level operation sequence is executed directly, reducing inference steps. If the high-level action is not applicable, it falls back to the low-level action space to ensure the task can still be completed.

AppAgentX project address

• Project official website:https://appagentx.github.io/
• GitHub repository:https://github.com/Westlake-AGI-Lab/AppAgentX
• arXiv technical paper:https://arxiv.org/pdf/2503.02268

Application scenarios of AppAgentX

• Automating daily operationsIt automatically completes phone settings adjustments and in-app tasks, reducing manual operations.
• Smart Assistant EnhancementIt can be integrated into a smart assistant to help users quickly perform complex tasks.
• Enterprise Process AutomationIt is used for repetitive tasks such as data entry and report generation in enterprises, improving efficiency.
• Cross-application task managementIt supports switching and operating between different applications, enabling cross-platform automation.
• Assisting special groups: Simplify the operation process to help the elderly or people with disabilities use mobile phones more easily.