project
AndroidGen - Zhipu launches a framework to enhance the capabilities of large language model agents.
AndroidGen is a framework developed by the Zhipu Technology team to enhance the capabilities of Large Language Model (LLM)-based agents, especially in situations where data is scarce. The framework collects human task trajectories and trains language models based on these trajectories...
What is AndroidGen?
AndroidGen is a framework developed by the Zhipu Technology team to enhance the capabilities of Large Language Model (LLM)-based agents, especially in situations where data is scarce. The framework collects human task trajectories and trains language models based on these trajectories, developing agents that do not require manual trajectory annotation, significantly improving the ability of LLMs to perform complex tasks.
Main functions of AndroidGen
- Data collection and training without manual annotationAndroidGen can develop efficient agents by collecting human task trajectories and training language models based on these trajectories without requiring manual trajectory annotation.
- Enhance the Agent's task execution capabilitiesThrough four core modules (ExpSearch, ReflectPlan, AutoCheck, and StepCritic), AndroidGen significantly enhances the ability of LLM to perform complex tasks.
- xpSearch (Experience Search)By retrieving similar completed trajectories, the LLM performs contextual learning, thereby improving the Agent's capabilities and helping it generalize from simple tasks to complex ones.
- Reflect Plan: Reflect on the current environment and update the plan status to enhance the Agent's long-term reasoning ability.
- AutoCheckActively verify the validity of each Agent operation to reduce the risk of task failure due to operational errors.
- StepCritic (Step-by-Step Evaluation)It breaks down the task into multiple sub-objectives and provides stepwise trajectory evaluation, offering fine-grained labels for model optimization.
- High-efficiency data collection pipelineAndroidGen builds an efficient data collection pipeline that generates a large number of high-quality Android browsing tracks.
The technical principles of AndroidGen
- Model trainingThe LoRA technique was used to fine-tune GLM-4-9B and Llama-3-70B on an automatically constructed dataset to obtain the Android Agent model. No manual trajectory annotation was required; by treating each step in the trajectory as an independent sample for training, the model fully utilized the information in the dataset.
- Hybrid planning and execution stepsThe planning and execution steps are combined and fine-tuned to enable LLM to have both planning and execution capabilities.
- Data collection process:
- Task formulationBased on GPT-4o, it generates approximately 300 task instructions from the instructions in AndroidWorld.
- Agent samplingThe trajectory of each task is sampled based on AndroidWorld and GPT-4o.
- Track recordingRecord the environment and operation information for each step to build a reproducible Android navigation trajectory.
- Trajectory EvaluationUse StepCritic to evaluate the recorded trajectory to ensure that each sub-objective has been completed.
- Trajectory Augmentation: Expanding the high-quality dataset, a dataset containing more than 1,000 trajectories was eventually built.
AndroidGen performance
- AndroidWorld Benchmark:
- AndroidGen significantly enhances the capabilities of the same base model Agent, with a more pronounced performance improvement compared to M3A and SeeAct.
- The AndroidGen + GPT-4o combination achieved an average score of 46.8, far exceeding other combinations.
- The average score of GLM-4-9B + AndroidGen, which has smaller model parameters and is open source, surpasses that of GPT-4o + M3A, which has larger model parameters and is closed source.
- AitW (Android in the Wild) In tests against eight globally popular mobile applications (such as Google Maps and YouTube), AndroidGen also performed exceptionally well in terms of its ability to understand and interact with natural language commands in real-world device environments.
Application scenarios of AndroidGen
- Automated task processingThrough natural language commands, the agent can automatically complete tasks such as sending emails, setting reminders, and querying information.
- Cross-application operationAgents can interact between different applications and perform operations such as copying data from one application to another.
- Intelligent NavigationOn Android devices, the Agent can navigate based on user commands, such as opening specific applications or searching for files.
- Intelligent InteractionThrough natural language understanding, agents can interact with users and provide a more intelligent user experience.