Open-AutoGLM - Zhipu's open-source mobile AI agent framework

What is Open-AutoGLM?

Open-AutoGLM is an open-source mobile intelligent assistant framework developed by Zhipu. Built on AutoGLM, it automates mobile phone operations through natural language commands. Open-AutoGLM uses the Phone Use framework to translate natural language commands into actual actions such as clicking, swiping, and typing, enabling automated task completion in scenarios such as food delivery, social networking, and customer service. The framework leverages cloud phone technology to ensure privacy and security. The purpose of Open-AutoGLM's open-source release is to promote industry progress, protect user privacy, and accelerate the development of agent technology. Currently, it supports over 50 mainstream Chinese applications, providing a complete toolchain and detailed documentation to help developers quickly get started and perform secondary development.

Main functions of Open-AutoGLM

• Natural Language Understanding and Task ExecutionUsers can describe their needs in natural language, and AI will automatically interpret the intent and execute the operation.
• Multimodal Interface UnderstandingIt uses a visual language model to understand the content on the mobile phone screen and recognize interface elements for operation.
• Automated operationIt supports various operations such as clicking, swiping, text input, long press, and double-tap, simulating real user behavior.
• Sensitive operation confirmation and manual takeoverWhen privacy or sensitive operations are involved, the AI will request human confirmation or take over to ensure security.
• Remote debugging and controlIt supports remote ADB debugging via WiFi or network, and can be operated without physical device connection.
• Supports multiple applicationsIt covers more than 50 mainstream Chinese applications, including social communication, e-commerce shopping, food delivery, video entertainment and other scenarios.
• Cloud phone environment deploymentIt runs in a virtual device in the cloud, ensuring user privacy and data security.

How to use Open-AutoGLM

• Preparation
  • Install PythonEnsure that Python is installed on your computer (version 3.10 or above is recommended).
  • Install ADBDownload and install ADB from https://developer.android.com/studio/releases/platform-tools. After installation, add the ADB path to your system environment variables.
  • Configure Android devicesEnsure that developer mode and USB debugging are enabled on your Android device and that it is connected to your computer.
• Download and install
  • Download Open-AutoGLM:
    • Visit the Open-AutoGLM GitHub repository: https://github.com/zai-org/Open-AutoGLM.
    • Click “Code” in the upper right corner of the page" Button, select "Download ZIP""Download the project code.
    • Extract the downloaded file to a convenient directory on your local machine (e.g., ...). C:\Open-AutoGLM or ~/Open-AutoGLM).
  • Install dependencies:
    • Open a command-line tool (Command Prompt or PowerShell on Windows, Terminal on Mac/Linux).
    • Switch to the unzipped project directory:
    • cd path/to/Open-AutoGLM
    • Enter the following command to install dependencies:
    • pip install -r requirements.txt
  • Start the model service:
    • Enter the following command to start the model service (this is a necessary step, but you do not need to understand the code):
    • python3 -m vllm.entrypoints.openai.api_server --model zai-org/AutoGLM-Phone-9B --port 8000
    • If you see a message indicating that the service started successfully, you can proceed to the next step.
• Run taskOpen a new command line window and enter the following command to execute the task (only the task description needs to be replaced):python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "任务描述"

Open-AutoGLM project address

• GitHub repositoryhttps://github.com/zai-org/Open-AutoGLM
• HuggingFace model libraryhttps://huggingface.co/zai-org/AutoGLM-Phone-9B

Application scenarios of Open-AutoGLM

• Food deliveryUsers only need to describe their needs in natural language, and Open-AutoGLM can automatically open the Meituan app, search for KFC, select a family bucket meal, and complete the order.
• Social media interactionIn social applications such as WeChat and Weibo, users can instruct AI to "like a friend's latest post" or "comment on a Douyin video." The AI will automatically recognize the relevant content and perform interactive operations such as liking and commenting.
• Office AutomationIn office software (such as WPS, Microsoft Office), users can use voice or text commands to "create a document named 'Project Plan' and write meeting content," and AI can complete the document creation and content editing.
• Smart Home ControlThrough smart home applications (such as Xiaomi Smart Home), Open-AutoGLM can accurately identify and control corresponding devices, enabling automated switching of home scenes.
• TransportationIn map or ride-hailing apps (such as Gaode Maps and Didi Chuxing), AI will plan routes in real time and complete the ride-hailing operation, which is convenient and fast.