ScreenAI - Google's readable screen AI visual model that can understand UI and infographics.

What is ScreenAI?

ScreenAI is a readable screen AI visual language model developed by Google researchers, specifically designed to understand and process user interfaces (UIs) and infographics. Based on the PaLI architecture, it combines visual and language processing capabilities and borrows Pix2Struct's flexible tiling strategy, enabling it to understand and generate text related to screen UI elements such as question answers, UI navigation instructions, and content summaries.

• arXiv research paper:https://arxiv.org/abs/2402.04615
• GitHub PyTorch implementation:https://github.com/kyegomez/ScreenAI

ScreenAI's main functions

• Screen Information UnderstandingScreenAI can recognize and understand the content of UI elements and infographics, including their type, location, and relationships with each other.
• Question and Answer (QA)ScreenAI can understand the visual information it receives and answer questions about the UI and infographic content.
• UI NavigationScreenAI can interpret navigation commands (such as "back") and recognize appropriate UI elements for interaction, understanding user intent and navigating accurately within the interface.
• SummaryScreenAI can concisely summarize screen content, extracting and summarizing the core points of screen information.
• Adapt to different screen formatsScreenAI can handle screenshots with different resolutions and aspect ratios, and can adapt to the screen formats of different devices such as mobile devices and desktops.

ScreenAI's technical principles

• Multimodal encoderInspired by the PaLI architecture, ScreenAI uses a multimodal encoder block consisting of two main parts: a visual encoder and a language encoder. The visual encoder, based on the Vision Transformer (ViT) architecture, transforms the input screenshot into a series of image embeddings. The language encoder processes textual information associated with the screenshot, such as labels and descriptions of user interface (UI) elements.
• Image and text fusionIn multimodal encoders, image embeddings and text embeddings are combined so that the model can simultaneously understand visual content and associated linguistic information. This fusion enables ScreenAI to handle complex screen interaction tasks.
• Autoregressive decoderThe encoder's output is passed to an autoregressive decoder T5, which is responsible for generating text output and can generate natural language responses based on the input image and text embeddings.
• Automatic data generationTo train ScreenAI, researchers utilized automated data generation techniques. They used the PaLM 2-S language model to generate synthetic training data, which included screen patterns and corresponding question-answer pairs. This approach increased the diversity and complexity of the data while reducing reliance on manual annotation.
• Image segmentation strategyScreenAI employs the Pix2Struct technique to process screenshots with varying resolutions and aspect ratios. This technique allows the model to generate image patches of arbitrary grid shapes based on the shape of the input image and a predefined maximum number of blocks, enabling the model to adapt to various screen formats.
• Model configuration and trainingScreenAI offers model versions at different scales, including 670M, 2B, and 5B parameter models. These models use different starting points during the pre-training phase, such as starting with the PaLI-3 multimodal pre-training checkpoint. The combination of pre-training and fine-tuning tasks allows the model to be trained and optimized on a wide range of tasks.