AB
AB
AiBoss

All Categories

project

CogAgent-9B - The base model for Zhipu AI's open-source GLM-PC

CogAgent-9B is a dedicated agent task model trained on GLM-4V-9B. It only requires screenshots as input and does not require text representations such as HTML. CogAgent-9B supports high-resolution image processing and is bilingual (Chinese and English)...

What is CogAgent-9B?

CogAgent-9B is a dedicated agent task model trained on GLM-4V-9B. It only requires screenshots as input and does not require text representations such as HTML. CogAgent-9B supports high-resolution image processing, has bilingual (Chinese and English) interaction capabilities, and can predict and execute GUI operations to automate tasks. It has achieved leading results on multiple GUI operation datasets, and the model is open source, driving the development of the large-scale agent model ecosystem. CogAgent-9B can be widely used in GUI interaction scenarios such as personal computers, mobile phones, and in-vehicle systems.

Main functions of CogAgent-9B

  • Understanding and operating the GUICogAgent-9B can understand and operate graphical user interfaces (GUIs), performing tasks such as clicking buttons and entering text.
  • Screenshot inputThe model relies solely on screenshots as input, requiring no additional text representations such as HTML, making it flexible for use on multiple devices.
  • High-resolution processingIt supports high-resolution image input up to 1120×1120 pixels, enabling it to process more complex visual information.
  • Bilingual InteractionIt supports screenshots and language interaction in both Chinese and English, enhancing the model's international application capabilities.
  • Predicting GUI operationsBased on the user-specified task and historical operations, the model can predict the next GUI operation.
  • Automated task executionCogAgent-9B can simulate user operations and automate a series of GUI tasks.
  • cross-platform applicationsIt is suitable for various GUI-based interaction scenarios such as personal computers, mobile phones, and in-vehicle systems.
  • Leading performanceIt achieved leading results on multiple GUI operation datasets, demonstrating its superior performance.

Technical Principles of CogAgent-9B

  • Visual Language Model (VLM)CogAgent-9B is built on top of the powerful visual language model GLM-4V-9B, enabling it to process visual data (such as screenshots) and text information, and to understand and manipulate GUI elements.
  • Two-stream attention mechanismCogAgent-9B employs a two-stream attention mechanism, mapping visual elements (such as buttons and icons) to their text labels or descriptions, enhancing the model's ability to predict user intent and perform related actions.
  • GUI Grounding Pre-trainingDuring the pre-training phase, CogAgent-9B introduces a GUI Grounding pre-training method, which constructs the correspondence between interface sub-regions and layout representations through screenshots and layout pairs, thereby improving the model's basic understanding of visual input and GUI interfaces.
  • Data sets are rich and completeThe CogAgent-9B team extensively collected and integrated various datasets, including unsupervised data and GUI instruction fine-tuning datasets. The richness and diversity of these datasets provided a broad foundation for training and testing the model.
  • Pre-training and post-training strategy optimizationCogAgent-9B introduces GUI Referring Expression Generation (REG) and GUI Referring Expression Comprehension (REC) tasks during the pre-training phase to construct the correspondence between interface sub-regions and layout representations. In the post-training phase, a more scientific GUI agent post-training strategy is adopted, enabling the model to possess stronger analytical, reasoning, and predictive capabilities.
  • Model reasoning and mental chain optimizationCogAgent-9B breaks down the thought chain into Status (current screen state), Plan (global plan), Action (next natural language description), and Operation (next formal language description), and flexibly adjusts and controls the actual output during the inference process by randomly sampling and mixing multiple modes of training data.
  • Improved movement spaceCogAgent-9B clarifies the basic action space and adds advanced actions such as LLM, QUOTE_TEXT, and LAUNCH, enhancing the model's usability and interactive capabilities.

CogAgent-9B project address

Application scenarios of CogAgent-9B

  • Automated testingIn software development, CogAgent-9B can be used for automated testing, simulating user actions to test the application's GUI, thereby improving testing efficiency and coverage.
  • Smart AssistantAs an intelligent personal assistant, CogAgent-9B can help users automate daily tasks such as schedule management and email processing.
  • Customer ServiceIn the field of customer service, CogAgent-9B can assist customer service personnel through automated operations, enabling them to quickly respond to customer needs and perform related actions.
  • Smart Home ControlThe CogAgent-9B can be integrated into smart home systems, allowing users to control various smart devices in their homes via a GUI.
  • Smart CockpitIn the automotive field, CogAgent-9B can be used in smart cockpit systems, interacting with in-vehicle infotainment systems through a GUI to provide a safer and more convenient driving experience.
Back