AB
AB
AiBoss

All Categories

project

MAI-UI - A full-size GUI intelligent agent base model open-sourced by Alibaba Tongyi

MAI-UI is a full-scale GUI intelligent agent foundation model launched by Tongyi Labs, possessing three core capabilities: user interaction, tool invocation, and edge-cloud collaboration. The framework utilizes an autonomously evolving data pipeline and large-scale online reinforcement learning technology...

What is MAI-UI?

MAI-UI is a full-scale GUI intelligent agent foundation model launched by Tongyi Labs, possessing three core capabilities: user interaction, tool invocation, and edge-cloud collaboration. Through its self-evolving data pipeline and large-scale online reinforcement learning technology, the framework achieves full-scale coverage from 2B to 235B-A22B, suitable for various scenarios. It leads in GUI visual positioning and task execution capabilities, topping multiple authoritative benchmark sets. MAI-UI's edge-cloud collaborative architecture ensures privacy while improving performance, and its dynamic environmental adaptability makes it perform exceptionally well in real-world tasks, providing strong support for next-generation human-computer interaction.

Main functions of MAI-UI

  • User InteractionWhen user instructions are incomplete or ambiguous, MAI-UI can proactively ask questions to clarify key information and ensure that task execution matches the user's true intentions.
  • Tool Call (MCP)It simplifies complex UI operations into efficient and reliable API calls by directly calling external tools through the Model-Callable Protocol (MCP).
  • End-to-cloud collaborationThe local lightweight model handles daily tasks, while complex tasks can be seamlessly switched to the cloud model, ensuring user privacy and data security.
  • Task executionEnables efficient GUI task automation across multiple operating systems, supporting cross-application operations and task planning in complex scenarios.
  • Dynamic environmental adaptabilityFaced with interference from pop-ups, advertisements, UI changes, etc. in the real environment, MAI-UI can autonomously go back or reposition itself to ensure task continuity.

MAI-UI's technical principles

  • Autonomous Evolutionary Data PipelineMAI-UI employs a self-evolving data pipeline, integrating multi-dimensional data such as user interactions and MCP tool calls into the training process. Through manual annotation and automatic model generation of high-quality training data, the data pipeline is continuously updated, enabling the model to evolve in complex scenarios and improve its understanding and execution capabilities for real-world tasks.
  • Massive online reinforcement learningThe model enhances generalization and robustness through large-scale online reinforcement learning. Training is performed in dynamic environments, supporting ultra-long trajectories (up to 50 steps) and dynamic perturbation injection mechanisms (such as pop-ups, permissions, and UI offsets). This enables the model to adapt to various disturbances in real-world GUI tasks, ensuring consistent task execution and high success rates.
  • End-to-cloud collaborative architectureMAI-UI employs an edge-cloud collaborative architecture, with a lightweight local model acting as a "trajectory monitor" to determine in real time whether the task execution path deviates from the user's intent. When a task gets stuck on the device side and does not involve privacy information, the system triggers the cloud model to take over, ensuring the task is completed smoothly. Meanwhile, operations involving privacy are always executed locally, protecting user data security.
  • Multimodal interaction capabilitiesBased on a multimodal large language model (such as Qwen3-VL), MAI-UI achieves visual perception and language understanding of the GUI interface. The model supports various operations, such as clicking, swiping, and input, and can complete complex task sequences, thereby achieving efficient GUI task automation on various operating systems (such as mobile phones, computers, and web pages).

MAI-UI project address

  • GitHub repositoryhttps://github.com/Tongyi-MAI/MAI-UI
  • HuggingFace model library:https://huggingface.co/Tongyi-MAI/models
  • arXiv technical paperhttps://arxiv.org/pdf/2512.22047

Application Scenarios of MAI-UI

  • Family life sceneWhen shopping for family items, MAI-UI can proactively remind users to add items such as cherries and laundry detergent to their shopping cart based on their calendar to-do list, ensuring that users do not forget important items.
  • Office SceneIn the office, assist users with document processing, such as finding resume files and sending them to HR colleagues, while proactively asking for key information to ensure the task is completed smoothly.
  • Travel scenariosUsers can ask MAI-UI to plan the optimal route from one location to another and record the results in notes for easy reference at any time.
  • social scenesSynchronize important information in social groups, such as arrival time, and @ relevant people in the group to ensure timely delivery of information.
  • Learning and Education ScenariosUsers can ask MAI-UI to open the learning platform, find a specific course, and record the key points of the course for later review.
Back