AB
AB
AiBoss

All Categories

project

ShowUI - National University of Singapore and Microsoft jointly launch a visual-language-operation model for GUI automation.

ShowUI is a visual-language-action model jointly developed by the Show Lab at the National University of Singapore and Microsoft, designed to improve the efficiency of graphical user interface (GUI) assistants. The model reduces computational costs based on UI-guided visual token selection...

What is ShowUI?

ShowUI, a vision-language-action model jointly developed by the Show Lab at the National University of Singapore and Microsoft, enhances the efficiency of graphical user interface (GUI) assistants. The model reduces computational costs through UI-guided visual token selection, unifies diverse requirements in GUI tasks with interleaved vision-language-action flows, and manages vision-action history to improve training efficiency. Using a small but high-quality instruction-following dataset, ShowUI achieved 75.1% zero-shot screenshot localization accuracy with only 256K data points, and improved training speed by 1.4 times, demonstrating its potential in the field of GUI visual agents.

ShowUI's main functions

  • UI-guided visual token selectionThe screenshots are used to construct a UI connection graph, which adaptively identifies redundant relationships and uses them as the criterion for selecting tokens in the self-attention module, reducing computational costs.
  • Interlaced visual-linguistic-motor flowIt flexibly unifies diverse requirements in GUI tasks, effectively manages vision-action history, and improves training efficiency.
  • Small-scale, high-quality GUI instruction follow datasetBy carefully curating data and employing resampling strategies to address data type imbalance, the accuracy and efficiency of the model can be improved.
  • Zero-sample screenshot positioningThe ability to directly understand and manipulate screenshots without additional training.
  • GUI AutomationAutomating GUI tasks, such as clicking and inputting, improves the efficiency of human-computer interaction.

The technical principles of ShowUI

  • UI-guided visual token selection:
    • The screenshot is divided into regular patches, with each patch serving as a node.
    • Identify adjacent patches with the same RGB values, construct a UI connection graph, and group together visually redundant areas.
    • In the self-attention module, visual tokens are selectively processed based on the UI connectivity graph, reducing computational load.
  • Interlaced visual-linguistic-motor flow:
    • Structured GUI actions, represented in JSON format, unify actions across different devices.
    • It manages complex interaction histories by alternating the processing of visual, linguistic, and behavioral data.
    • During training, a multi-turn dialogue approach is used to improve data utilization efficiency.
  • Data planning and resampling strategies:
    • Carefully plan and select high-quality training data, rather than simply aggregating all available data sources.
    • Based on a resampling strategy, the problem of data imbalance between different devices and task types is solved.
  • Efficiently process high-resolution UI screenshotsFor high-resolution UI screenshots, the model is optimized to effectively handle long token sequences and reduce computational costs.
  • Model Architecture:
    • Based on the Qwen2-VL-2B model, a visual encoder and a language model are integrated to process visual and text data.
    • Improve model performance in GUI tasks by using specific data recipes and training strategies.

ShowUI's project address

Application Scenarios of ShowUI

  • Web page automationAutomatically execute clicks, inputs, scrolling, and other operations on web pages, used in automated testing, data scraping, or simulating user behavior.
  • Mobile application testing: To automate various user interactions in mobile applications, such as swiping, clicking, and filling out forms, and to test application functionality.
  • Desktop software automationAutomating repetitive tasks in desktop software, such as file management, data entry, and settings adjustments.
  • Virtual AssistantAs part of a virtual assistant, it performs specific GUI operations based on the user's natural language commands.
  • Game AutomationIn games that support automated scripts, it can automatically perform actions such as character movement, item pickup, and combat.
Back