AndroidLab - Tsinghua University and Peking University jointly launch a framework for systematically evaluating Android intelligent agents.

What is AndroidLab?

AndroidLab is a framework for training and evaluating autonomous Android agents. It integrates text and image modal manipulation environments, unifies the action space, and provides reproducible benchmarks. AndroidLab supports large language and multimodal models, includes 138 tasks covering nine applications, and allows for the development of Android instruction datasets to improve the success rate of open-source models. The framework narrows the performance gap between open-source and closed-source models, promotes the development of open-source solutions, and is open-source on GitHub.

Main functions of AndroidLab

• Multimodal operating environmentIt provides a standardized multimodal operating environment, supporting large language models (LLMs) and multimodal models (LMMs) to interact within the same operating space.
• BenchmarkingDesign a benchmark test with 138 tasks covering nine common applications. The task types include operation tasks and query tasks, involving complex real-world interaction scenarios.
• Evaluation indicatorsThe system introduces evaluation metrics such as task completion rate (SR), sub-target success rate (Sub-SR), reverse redundancy rate (RRR), and reasonable operation ratio (ROR) to accurately assess the agent's task performance.
• Dataset ConstructionCreate an Android Instruct dataset, combine automated exploration with manual annotation to generate high-quality operational data, and improve the performance of open source models in terms of task completion rate and operational efficiency.
• Model training and optimizationIt supports training and optimizing both open-source and closed-source models, and significantly reduces the performance gap between open-source and closed-source models based on instruction tuning.

AndroidLab Technical Principles

• Operating mode:
  • XML schemaDesigned for text input LLMs, based on compressed XML information transmission interface state, the model directly selects elements to perform operations.
  • SoM modeFor LMMs, the Set-of-Mark method is used to combine screenshots with marker information, and the model selects the numbered marker elements for operation.
• Reasoning and Operational Framework:
  • ReAct frameworkBased on step-by-step reasoning and operation output, the model outputs the operation while displaying the intermediate reasoning process.
  • SeeAct frameworkReasoning and operation are separated and carried out in two rounds of interaction. In the first round, the model generates detailed reasoning steps, and in the second round, specific operations are executed.
• Task design and reproducibilityEach task is designed with multiple sub-goals, and XML tree structure matching is used to verify the completion of each sub-goal, ensuring the accuracy of the results and the efficiency of the operation.
• Evaluation SystemThe task completion-based evaluation system directly judges from the device and screen status, providing a comprehensive and accurate assessment of agent performance.
• Datasets and Model TrainingThe dataset is built based on automated exploration and manual annotation, and the model is trained and fine-tuned to improve the model's performance in multimodal and text input tasks.

AndroidLab project address

• GitHub repository:https://github.com/THUDM/Android-Lab
• arXiv technical paper:https://arxiv.org/pdf/2410.24024

AndroidLab Application Scenarios

• Automated testingUsed for automated testing of mobile applications, simulating user actions to test the application's functionality and performance.
• Smart Assistant DevelopmentDevelop intelligent assistants, such as voice assistants and chatbots, that can understand and execute complex user commands.
• Human-computer interaction researchResearch and improve human-computer interaction methods, especially on mobile devices, on how to enable machines to better understand and respond to human commands.
• Artificial intelligence researchIt provides an experimental platform for researching and developing the performance of large language models (LLMs) and multimodal models (LMMs) in practical applications.
• Education and trainingAs an educational tool, it helps students and researchers understand how the Android operating system works and how to develop applications.