Qwen-UI-Agent - A GUI intelligent agent foundation model launched by Alitongyi

What is Qwen-UI-Agent?

Qwen-UI-Agent is a GUI intelligent agent foundation model developed by Alibaba's Tongyi team. It enables users to interact with interfaces on mobile phones, computers, and web browsers much like a real person. The model was trained on over 100 real devices and 150+ applications, supports mixed execution of GUI and command lines, and can complete long-range tasks across applications. It has built-in security mechanisms that directly reject dangerous requests and proactively require user confirmation for sensitive operations such as payments and deletions. It comprehensively surpasses flagship models like GPT-5.6 and Claude Opus on benchmarks such as MobileWorld and WebArena, making it one of the most usable GUI agents in the real world today.

Main functions of Qwen-UI-Agent

• Cross-device GUI operationAutomatically click, input, and swipe on mobile phones, computers, and browsers to complete interface interactions.
• Real device multi-application workflowExecute complex, long-chain tasks across 150+ real apps, such as checking maps → finding restaurants → making reservations → sending schedules.
• Hybrid execution of GUI + CLIIt can be used both as a user interface and as a command line, and supports outputting multiple batch actions at once to speed up the process.
• Security Boundary ControlIllegal requests will be rejected outright, while sensitive operations such as payments, file deletion, and privacy authorization will be proactively prompted to prompt the user for confirmation via pop-up windows.
• Long-term task processingIt can reliably execute long tasks such as research, data analysis, and report generation, which involve more than 100 steps.
• Precise interface positioningIt refreshes the state-of-the-art (SOTA) benchmark on grounding standards such as ScreenSpot-Pro, accurately identifying the position of screen elements.

The technical principles of Qwen-UI-Agent

• Real machine training environmentBuild a real device cluster of 100+ real mobile phones and 150+ applications to directly collect real trajectories for training and evaluation, bridging the performance gap between simulated environments and real devices.
• Hybrid motion spaceThe model simultaneously masters GUI operations (click, swipe, input) and CLI commands, and introduces a Batched Actions mechanism to support multiple actions output in parallel in a single decision, making CLI account for nearly half of the computer tasks and significantly shortening the overall execution steps.
• Long-term online reinforcement learningIt supports end-to-end online RL training on ultra-long trajectories of more than 100 steps, relying on parallel rollout in approximately 10,000 concurrent environments, combined with adaptive course learning, to continuously tackle highly challenging long-range tasks.
• Safe alignment mechanismSecurity checks are embedded throughout the entire task execution chain, identifying illegal or high-risk requests through pre-classification and terminating them directly; for sensitive scenarios such as payments, data deletion, and privacy authorization, they are triggered at key nodes. ask_user Take over to ensure that dangerous requests are not executed and sensitive operations are not decided without authorization.
• Multimodal Grounding and General Capabilities PreservationWhile training specifically for GUI, it retains the general reasoning and agentic capabilities of the base model, enabling it to accurately locate screen elements (refreshing multiple grounding state-of-the-art) and meet the needs of long-tail knowledge and tool calls in the real world.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Qwen-UI-Agent

• Reading technical reportsVisit the arXiv paper to learn about the model architecture, training methods, and detailed evaluation results on 8 benchmarks.
• Browse the project homepageOpen tongyi-mai.github.io/Qwen-UI-Agent to watch 5 real-world device demos and interaction examples.
• Tracking open source repositoriesFollow GitHub Tongyi-MAI/MAI-UI to get updates on model weights, code, and deployment documentation that may be released later.
• Operating benchmark evaluationUsing the Docker environment of the accompanying repository Tongyi-MAI/MobileWorld, we conducted reproducible mobile GUI capability tests on the model.

Qwen-UI-Agent's core advantages

• Real machine closed loopBased on training and evaluation of 100+ real mobile phones and 150+ applications, we have built our own MobileWorld-Real real device benchmark, directly bridging the gap between "simulation and reality".
• Full SOTAIt surpasses or matches flagship models such as GPT-5.6, Claude Opus 4.8, and Gemini 3.1 Pro in all eight core benchmarks, including MobileWorld, OSWorld, WebArena, and ScreenSpot-Pro.
• Hybrid execution speedupUnified GUI clicks and CLI commands support batch output of multiple actions in a single decision. Approximately 40% of computer tasks are batch actions, saving nearly 58% of the execution steps overall.
• Long-term online reinforcement learningSupports end-to-end online RL for ultra-long trajectories of 100+ steps, relying on parallel rollout in a concurrent environment of approximately 10,000, continuously tackling complex long-range tasks.
• Built-in security boundaryIllegal/dangerous requests will be directly rejected and terminated; sensitive operations such as payment, data deletion, and privacy authorization will be proactively stopped at key points, and a pop-up window will be displayed to ask the user for confirmation before continuing.

Qwen-UI-Agent project address

• Project official website：https://tongyi-mai.github.io/Qwen-UI-Agent/
• GitHub repository：https://github.com/Tongyi-MAI/MAI-UI
• Technical Papers：https://tongyi-mai.github.io/Qwen-UI-Agent/Qwen-UI-Agent-Technical-Report.pdf

Comparison of Qwen-UI-Agent with similar competing products

Application Scenarios of Qwen-UI-Agent

• Local Lifestyle Store VisitsIt automatically searches addresses, compares popularity, and summarizes reviews across Gaode Maps, Dianping, and Xiaohongshu, making dating and restaurant exploration a one-stop solution.
• Itinerary and meeting coordinationCheck 12306 high-speed rail schedules, calculate subway commute times, and automatically create meeting schedules with reminders on DingTalk.
• In-depth research reportIt can collect financial data from multiple websites, run analysis scripts, and automatically generate Excel, PPT, and Word deliverables.
• Cross-device invoice organizationExtract invoice photos from your phone's photo album, transfer them to your computer for categorization and archiving, and generate a structured Excel expense report.
• Proactive emergency response to flight cancellationsUpon detecting a flight cancellation notification, the system automatically searches for alternative flights and high-speed rail options, recommends the best option that allows the user to make the meeting, and then returns the decision to the user.