Aria-UI - A multimodal model for intelligent GUI interaction, jointly developed by HKU and Rhymes AI.
Aria-UI is a large-scale multimodal model designed for graphical user interface (GUI) localization tasks, jointly developed by the University of Hong Kong and Rhymes AI. Based on a purely visual approach, it does not rely on auxiliary inputs such as HTML or AXTree, and utilizes large-scale...
What is Aria-UI?
Aria-UI is a large-scale multimodal model designed for graphical user interface (GUI) localization tasks, jointly developed by the University of Hong Kong and Rhymes AI. Based on a purely visual approach, it does not rely on auxiliary inputs such as HTML or AXTree. Using a large-scale, diverse data synthesis pipeline, it generates high-quality element descriptions and instruction samples from Common Crawl and publicly available data, adapting to diverse instructions in different environments. Aria-UI innovatively integrates dynamic action history information, using text or text-image interleaved formats to enhance the model's localization capabilities in multi-step task scenarios. In a wide range of benchmark tests, including offline and online agent tasks, Aria-UI achieved excellent performance, demonstrating strong zero-shot generalization ability and cross-platform applicability, making it a powerful solution for general GUI localization tasks.
Main functions of Aria-UI
- GUI element positioningIt accurately maps language commands to target elements in the GUI, enabling precise element positioning and providing a foundation for subsequent interactive operations.
- Multimodal input processingIt processes input information from multiple modalities, including GUI images, text commands, and action history, and fully understands and utilizes the rich information in multimodal data.
- Diverse instruction adaptationBased on a large-scale and diverse data synthesis process, it generates diverse instruction samples that effectively adapt to various instruction expression methods in different environments.
- Dynamic Context UnderstandingIt integrates dynamic action history information and uses text or text-image interleaved formats to understand the current dynamic context in multi-step task scenarios, providing an important reference for accurate element location.
- High-resolution image processingSupports image resolutions up to 3920×2940, segments images into smaller blocks for processing, and significantly expands the range of image sizes that can be processed.
Aria-UI's technical principles
- Purely visual methodsIt employs a purely visual approach to directly extract visual features from GUI images, using visual information to understand and locate target elements.
- Multimodal MoE modelIt is built based on the Aria multimodal MoE (Mixture of Experts) model, has 3.9B activation parameters, and is good at processing multimodal data.
- Data synthesis and trainingBased on a two-stage data synthesis process, high-quality element descriptions and diverse instruction samples are generated from Common Crawl and public data, covering three major GUI environments: Web, desktop, and mobile. This provides a large amount of diverse training data for model training, enhancing the model's ability to recognize different instructions and elements.
- Context-Aware Data ExtensionThe model simulates contextualized localization tasks using publicly available proxy trajectory data, constructs two context settings: text action history and text-image interleaved history, and generates detailed step-by-step instructions for all localization steps in the trajectory data based on the data synthesis process, enabling the model to better understand and execute tasks in dynamic environments.
- Super-resolution supportIt segments and processes images into smaller pieces, supporting image resolutions up to 3920×2940 while maintaining image detail and accuracy.
Aria-UI project address
- Project official website:https://ariaui.github.io/
- GitHub repository:https://github.com/AriaUI/Aria-UI
- HuggingFace model library:https://huggingface.co/Aria-UI
- arXiv technical paper:https://arxiv.org/pdf/2412.16256
Application Scenarios of Aria-UI
- Automated testingWeb application testing automatically clicks webpage buttons and inputs information to verify whether the functions are working properly.
- User interaction assistanceVoice commands can be used to control home appliances, such as automatically clicking the switch button when you say "turn on the light".
- Intelligent Customer ServiceE-commerce platform customer service can quickly locate product details and answer user inquiries.
- Education industry: Automatically operates the code editor, demonstrating the programming process and results.
- Automated officeAutomated operation of financial software, generating reports, and improving work efficiency.