AB
AB
AiBoss

All Categories

project

PC Agent - A computer intelligence AI system jointly launched by Shanghai Jiao Tong University and GAIR

PC Agent is an advanced AI system jointly developed by Shanghai Jiao Tong University and the Generative AI Research Lab (GAIR). Based on simulating human cognitive processes, the system performs tasks such as organizing research materials, drafting reports, and creating presentations...

What is PC Agent?

PC Agent is an advanced AI system jointly developed by Shanghai Jiao Tong University and the Generative AI Research Lab (GAIR). Based on simulating human cognitive processes, the system performs complex digital tasks such as organizing research materials, drafting reports, and creating presentations. PC Agent integrates PC Tracker for efficient collection of human-computer interaction data, using a two-stage cognitive process to transform raw data into cognitive trajectories. Employing a multi-agent system architecture, PC Agent combines planning and localization agents to achieve accurate visual positioning and decision-making. Trained on a limited amount of high-quality cognitive data, the system can handle complex workflows of up to 50 steps, demonstrating exceptional data efficiency and practical application potential.

Main functions of PC Agent

  • Task automationAutomating complex digital tasks, such as organizing research materials, drafting reports, and creating presentations.
  • Human-computer interaction data collectionBased on PC Tracker, it collects detailed trajectories and cognitive contexts of user interactions with computers.
  • Cognitive trajectory transformationThe process is completed using two stages of cognition, transforming raw interaction data into trajectories rich in cognitive information.
  • Complex task processingHandling complex workflows involving multiple applications, such as switching between PowerPoint and a browser to collect information.
  • Multi-agent collaborationBy combining planning agents and positioning agents, decision-making and precise visual positioning can be achieved.
  • Training with a small amount of data: To execute complex workflows with only a small amount of cognitive trajectory data for training.

Technical Principles of PC Agent

  • PC Tracker:
    • Data collectionIt runs in the background, records the user's keyboard and mouse activities, captures screenshots, and collects human-computer interaction data.
    • Event TrackingAn event-based tracking strategy records key user action events, rather than continuous video streams, reducing storage requirements.
    • Unified motion spaceEncapsulate keyboard and mouse operations into a unified action space to simplify AI's understanding of human behavior.
  • Cognitive Completion Process:
    • Data RefinementBased on trajectory filtering, action filtering, and standardization, the quality of raw interaction data is optimized.
    • Action semantic completion: To supplement semantic information for click-related actions and generate a high-quality description of the click target.
    • Reconstruction of thought processesBased on action semantic information, reconstruct the implicit reasoning process behind each action.
  • Multi-agent system:
    • Planning Intelligent Agent: Responsible for making action decisions and gaining effective planning capabilities based on learning human cognitive trajectories.
    • Location agentIt is responsible for executing click-related actions and has a self-verification mechanism, achieving near-human accuracy.
    • Error correction mechanismWhen the positioning agent discovers that the target the planning agent is trying to click does not exist on the screen, the planning agent will be prompted to revise its action plan.

PC Agent project address

Application scenarios of PC Agent

  • Office AutomationCreate and edit documents, spreadsheets, and presentations; automate routine office tasks; and improve work efficiency.
  • Research and AcademicIt organizes and analyzes research data, automatically generates literature reviews, and assists in writing academic papers.
  • Content creationAutomatic layout and formatting improve the efficiency and quality of content creation.
  • project managementAutomatically update project reports, coordinate team tasks, and optimize project management processes.
  • Customer ServiceManage customer data and provide fast, personalized customer support.
Back