AgentCPM-GUI - An open-source edge-side GUI intelligent agent model developed by Tsinghua University and Mianbi Intelligence.
AgentCPM-GUI is an open-source edge GUI agent jointly developed by Tsinghua University and the Mianbi AI team, optimized for Chinese applications. The model is built on MiniCPM-V (8 billion parameters) and can accept smartphone screenshots as input...
What is AgentCPM-GUI?
AgentCPM-GUI is an open-source edge GUI agent jointly developed by Tsinghua University and the Mianbi AI team, optimized for Chinese applications. The model is built on MiniCPM-V (8 billion parameters) and can accept smartphone screenshots as input, autonomously executing user-specified tasks. The model is pre-trained on large-scale Chinese Android application interface data, significantly improving its understanding and localization capabilities of GUI elements. AgentCPM-GUI achieves state-of-the-art performance on both the Chinese Grounding Benchmark and Agent Benchmark, making it the first open-source GUI agent finely optimized for Chinese applications.
Main functions of AgentCPM-GUI
- Chinese application operationIt enables users to understand and operate various Chinese applications, such as Amap, Dianping, Bilibili, and Xiaohongshu.
- Automated task executionAfter receiving user instructions, it automatically breaks down the task steps and executes them accurately in the corresponding application, such as ordering food or playing videos.
- High-quality GUI positioningAccurately locate GUI elements such as buttons, input boxes, and labels on the screen.
- OCR Positioning and RecognitionIt can identify the text content on the screen and perform corresponding operations based on the text description.
Technical Principles of AgentCPM-GUI
- Pre-trainingIt is pre-trained on large-scale Chinese Android application interface data, covering common GUI controls such as buttons, input boxes, labels, and icons. Based on high-quality GUI grounding pre-training, it improves the model's ability to understand and locate visual interface elements. It also has OCR grounding capabilities to accurately locate and recognize text content on the screen.
- Enhanced Fine-tuning (RFT)The design incorporates reward functions across three dimensions: action format reward, action type reward, and action parameter reward. These functions guide the model to autonomously generate high-quality thought processes. Guided by these reward functions, the model continuously optimizes its strategies to obtain higher rewards and improve task execution success rates.
- Compact motion space designBased on a compact JSON format, the average action length is compressed to 9.7 tokens, reducing computational and memory requirements during inference. The compact action space design makes the model more suitable for deployment on mobile devices, accelerating edge inference and mitigating privacy and security concerns.
Project address for AgentCPM-GUI
- GitHub repository:https://github.com/OpenBMB/AgentCPM-GUI
- HuggingFace model library:https://huggingface.co/openbmb/AgentCPM-GUI
Application scenarios of AgentCPM-GUI
- Smart AssistantIt helps users complete various application operations based on voice or text commands, such as ordering takeout and playing music.
- Automated testingUsed to automate the testing of Chinese application functions and interfaces, improving testing efficiency.
- Elderly careSimplify how seniors use smartphones, such as making video calls and checking the weather.
- Assistance for visually impaired peopleIt combines voice commands and a screen reader to help visually impaired people use the application more conveniently.
- Enterprise Application AutomationAutomating repetitive tasks in an enterprise, such as data entry and report generation, improves efficiency.