OmniParser - A screen parsing tool from Microsoft that converts UI screenshots into structured data.
OmniParser is a screen parsing tool developed by Microsoft Research that converts user interface screenshots into structured data. The tool is specifically designed to improve the performance of UI proxy systems based on large language models (such as GPT-4V)...
What is OmniParser?
OmniParser is a screen parsing tool developed by Microsoft Research that converts screenshots of user interfaces into structured data. The tool is specifically designed to improve the performance of UI agent systems based on large language models (such as GPT-4V), enhancing the agent's ability to perform tasks by accurately identifying interactive icons and understanding the semantics of screenshot elements. OmniParser uses a finely tuned model to extract the semantics of interactive areas and element functions in the screen, improving the accuracy of operations and agent performance in multiple benchmark tests without relying on additional information such as HTML or view hierarchies. OmniParser can be used with various state-of-the-art LLMs: OpenAI (4o/o1/o3-mini), DeepSeek R1, Qwen 2.5-VL, and Anthropic (Sonnet).
Main functions of OmniParser
- Analyzing UI screenshotsOmniParser parses screenshots of user interfaces into structured elements, including interactive icons and text.
- Identify interactive areasUse specialized detection models to identify and label interactive areas in the UI, such as buttons and icons.
- Extracting functional semanticsThe functional semantics of the detected elements are extracted using a descriptive model to generate descriptions related to user tasks.
- Enhance agent performanceBy combining the analysis results, the performance and accuracy of UI proxies based on large language models (such as GPT-4V) can be improved when performing tasks.
- Cross-platform applicationsIt supports multiple operating systems and applications, such as Windows, macOS, iOS, Android, and various web browsers and desktop applications.
- No additional information requiredIt can independently extract necessary data from visual screenshots without relying on additional information such as HTML or view hierarchy.
OmniParser's technical principles
- Dataset ConstructionExtract data from popular web pages and applications to build interactive icon detection datasets and icon description datasets.
- Model fine-tuning:
- Detection modelFine-tuning the YOLOv8 model on an interactive icon detection dataset to identify and locate interactive areas in the UI.
- Description ModelFine-tuning the BLIP-v2 model on the icon description dataset generates functional descriptions for the icons.
- OCR moduleThe integrated optical character recognition (OCR) module extracts text from the screen, merges it with icon detection results, and removes highly overlapping bounding boxes.
- Structured representationThe detected elements and generated descriptions are integrated to form a structured DOM (Document Object Model) representation, and a bounding box screenshot is overlaid on the potentially interactive elements.
- Action predictionBy combining structured elements and functional semantics, the burden on large language models in action prediction is reduced, and the predicted actions are more accurately translated into actual actions on the screen.
OmniParser's project address
- Project official website:https://www.microsoft.com/en-us/research/articles/omniparser
- GitHub repository:https://github.com/microsoft/OmniParser/
- HuggingFace model library:
- arXiv technical paper:https://arxiv.org/pdf/2408.00203
Application scenarios of OmniParser
- Automated software testingIn the software development process, it automatically identifies and manipulates UI elements, executes test scripts, and improves test efficiency and coverage.
- Virtual AssistantAs part of a virtual assistant, it helps understand screen content and perform tasks such as booking, querying, and data entry.
- assistive technologyFor visually impaired individuals, OmniParser can analyze screen content and convert it into speech output, providing screen reading functionality.
- User interface design verificationDesigners verify whether the interactive elements in the UI design meet the expected functions and semantics.
- Cross-platform application developmentDevelopers test and optimize the application's UI across different operating systems and devices to ensure a consistent user experience.