PC Agent-E - An intelligent agent training framework jointly developed by Shanghai Jiao Tong University and SII
PC Agent-E is a high-efficiency intelligent agent training framework jointly developed by Shanghai Jiao Tong University and SII. The framework uses 312 human-annotated computer usage trajectories to synthesize diverse action decisions based on the Claude 3.7 Sonnet model, significantly...
What is PC Agent-E?
PC Agent-E is a high-efficiency intelligent agent training framework jointly developed by Shanghai Jiao Tong University and SII. The framework uses 312 human-annotated computer usage trajectories to synthesize diverse action decisions based on the Claude 3.7 Sonnet model, significantly improving data quality. The framework comprises four key parts: trajectory collection, thought chain completion, trajectory enhancement, and agent training. PC Agent-E achieved a 241% performance improvement in the Windows AgentArena-V2 benchmark, surpassing the extended thinking mode of Claude 3.7 Sonnet, becoming the next-generation state-of-the-art (SOTA) open-source computer intelligent agent on Windows systems.
Main functions of PC Agent-E
- High-efficiency trainingWith only 312 human-annotated trajectories, data augmentation significantly improves model performance.
- Cross-platform generalizationIn OSWorld benchmark tests, it demonstrates strong cross-platform capabilities, making it suitable for different operating systems.
- Task executionIt supports completing a variety of complex tasks, such as file operations, software usage, and web browsing.
- Data AugmentationBased on synthetic diversified action decisions, rich trajectory data is obtained, and the generalization ability of the model is improved.
Technical Principles of PC Agent-E
- Trajectory CollectionThis method uses a PC Tracker tool to record human action trajectories, including task descriptions, screenshots, and keyboard/mouse operations. A small number of high-quality human action trajectories are collected using a simple annotation process.
- Thought CompletionThis feature adds underlying reasoning logic to each action step based on the Claude 3.7 Sonnet model. It generates a reasoning process that aligns with human thought processes, based on a task description, historical actions, and current state.
- Trajectory BoostThis approach synthesizes more action decisions for each step of the trajectory, capturing the diversity of the task. Using the Claude 3.7 Sonnet model, multiple reasonable action decisions are generated for each step, enriching the trajectory data.
- Agent TrainingThe model was trained based on the open-source model Qwen2.5-VL-72B. A simple end-to-end training framework was used to ensure that the model could learn and perform tasks efficiently.
- Evaluation and ValidationValidate model performance in WindowsAgentArena-V2 and OSWorld benchmark tests. Adjust the number of synthesized actions to verify the key role of trajectory augmentation methods in improving performance.
PC Agent-E project address
- Project official website:https://gair-nlp.github.io/PC-Agent-E/
- GitHub repository:https://github.com/GAIR-NLP/PC-Agent-E
- HuggingFace model library:https://huggingface.co/henryhe0123/PC-Agent-E
- arXiv technical paper:https://arxiv.org/pdf/2505.13909
Application scenarios of PC Agent-E
- Automated officeIt automatically completes tasks such as document editing and data analysis, improving office efficiency.
- Software testingSimulate user operations to identify errors and problems in the software and improve software quality.
- Educational SupportAs a virtual teaching assistant, it helps students complete computer operation tasks and provides real-time guidance.
- Assisting people with disabilitiesIt provides assistive operation functions to facilitate computer use by people with disabilities.
- Cross-platform compatibilityIt enables seamless migration and task execution between different operating systems.