AGUVIS - The University of Hong Kong and Salesforce jointly launched a unified, purely visual GUI automation framework.
AGUVIS is a unified, purely visual framework jointly developed by the University of Hong Kong and Salesforce. It is designed for autonomous GUI agents and can operate on various platforms, such as web, desktop, and mobile devices. AGUVIS is based on image observation and...
What is AGUVIS?
AGUVIS, a unified pure vision framework jointly developed by the University of Hong Kong and Salesforce, is designed for autonomous GUI agents and can operate on various platforms, such as web, desktop, and mobile devices. Based on image observation and the association between natural language commands and visual elements, AGUVIS uses a consistent action space to achieve cross-platform generalization. AGUVIS combines explicit planning and reasoning to enhance the agent's autonomous navigation and interaction capabilities in complex digital environments. Through large-scale datasets and a two-stage training process, the framework achieves performance exceeding existing methods in both offline and online scenarios, becoming the first pure vision GUI agent to independently complete tasks without relying on external closed-source models.
AGUVIS main functions
- Cross-platform autonomous GUI interaction: Autonomously execute GUI tasks on different platforms (such as websites, desktops, and mobile devices).
- Image observation and natural language command associationMapping natural language commands to visual interface elements enables image-based interaction.
- Explicit Planning and ReasoningIt integrates planning and reasoning capabilities, enabling agents to analyze the environment and generate effective operational procedures.
- Large-scale dataset constructionCreate a large-scale GUI agent trajectory dataset that includes multimodal inference and underlying principles.
AGUVIS Technical Principles
- Pure visual frameworkIt adopts a purely visual approach, unifying interface observation as images and basing instructions on image coordinates, thereby improving the generalization ability across environments.
- Unified Action SpaceUsing a standardized action space and plugin system, consistent learning and interaction are achieved across different platforms.
- Visual-Language Model (VLM)VLM serves as the foundation, such as Qwen2-VL, which processes high-resolution images of arbitrary resolution and dynamically converts them into visual tokens.
- Two-stage training paradigm:
- Phase 1: Basic Training: Focuses on enabling the model to understand and interact with objects in a single GUI screenshot.
- Phase Two: Planning and Reasoning TrainingBuilding upon basic training, we introduce more complex decision-making and reasoning processes, and train the model based on diverse agent trajectory data.
- Inner Monologue: Generate detailed introspective monologues during training, including observations, descriptions, reflections, and low-level action instructions, to simulate the agent's thought process and improve its planning capabilities.
- Plug-in systemIt provides flexibility for actions that cannot be directly mapped to the existing action space, allowing the model to adapt to new environments and tasks.
AGUVIS project address
- Project official website:aguvis-project
- GitHub repository:https://github.com/xlang-ai/aguvis
- arXiv technical paper:https://arxiv.org/pdf/2412.04454
Application scenarios of AGUVIS
- Automated testingIn software development, it automates the testing of various graphical user interfaces, including websites, desktop applications, and mobile applications, to ensure the stability and reliability of the software.
- Virtual AssistantAs a virtual assistant, it helps users automate daily computer tasks such as schedule management, email processing, and data entry.
- Business process automation: To automate specific business processes within an enterprise, such as financial report generation and customer data management, thereby improving work efficiency.
- Education and trainingTo create interactive educational software that simulates real-world GUI operations for teaching and training purposes.
- Customer service automationIn the field of customer service, it automatically processes customer requests and provides rapid responses and solutions based on a GUI interface.