AutoDroid-V2 - A mobile GUI automation script agent launched by Tsinghua University

What is AutoDroid-V2?

AutoDroid-V2, developed by the Tsinghua University Artificial Intelligence Industry Research Institute, is a mobile GUI automation script agent based on Small Language Models (SLM). It enhances on-device GUI agent performance through code generation technology. AutoDroid-V2 analyzes an application's exploration history, automatically generating detailed application documentation to guide the SLM in generating multi-step scripts to complete complex user tasks. Compared to traditional step-by-step GUI agents, AutoDroid-V2 significantly improves task completion rates and execution efficiency, reduces reliance on large cloud models, lowers user privacy risks, and reduces centralized service costs. AutoDroid-V2 performs exceptionally well in multiple benchmark tests, demonstrating its potential for comprehensive on-device GUI agent deployment.

Main functions of AutoDroid-V2

• Automated UI task executionIt automatically completes various UI tasks for users on mobile devices by generating and executing multi-step scripts, such as opening applications, entering text, and clicking buttons.
• Code generation and executionTransform the UI task automation problem into a code generation problem, generate executable code scripts using a small language model (SLM) on the device, and execute the scripts efficiently with a code interpreter.
• Application Documentation GenerationIt automatically analyzes the application's exploration history, generates detailed application documentation, records the application's GUI state, important elements and their interaction relationships, and provides guidance for script generation.

AutoDroid-V2 Technical Principles

• arXiv technical paper:https://arxiv.org/pdf/2412.18116

Application scenarios of AutoDroid-V2

• Daily life assistanceWhen users are busy, the system automatically replies to messages based on their content, such as work emails and greetings from friends, to avoid missing important information.
• Daily workAutomatically record meeting content, extract key information, and generate meeting minutes during the meeting to facilitate post-meeting review and execution of related tasks, thereby improving meeting efficiency.
• Application testingAutomatically execute regression test cases after application updates to check whether the new version of the application functions properly, promptly identify and fix potential bugs, and ensure the stability and reliability of the application.
• Smart HomeBased on the user's daily routine and the current ambient light, the system automatically adjusts the brightness and color temperature of the lights in the home, such as automatically dimming the lights at night to create a comfortable sleeping environment and improve the quality of life.
• Special industry applicationsAfter a patient is discharged from the hospital, follow-up reminders and health questionnaires are automatically sent to collect information on the patient's recovery, helping doctors to understand changes in the patient's condition in a timely manner and provide better follow-up treatment and rehabilitation guidance.