ScreenAgent - A computer-controlled intelligent agent based on a visual language model

What is ScreenAgent?

ScreenAgent is a computer-controlled intelligent agent developed by a research team at the School of Artificial Intelligence, Jilin University. Based on a Visual Language Model (VLM), this agent can interact with real computer screens. Researchers constructed a "plan-execute-reflect" workflow to guide the agent's continuous interaction with the computer screen. ScreenAgent's core function is to manipulate the graphical user interface (GUI) by observing screenshots and outputting corresponding mouse and keyboard actions, thereby performing complex, multi-step tasks.

ScreenAgent official website entrance

• Official GitHub repository:https://github.com/niuzaisheng/ScreenAgent
• Arxiv research paper access:https://arxiv.org/abs/2402.07945

ScreenAgent's operation process

• Screen ViewScreenAgent can observe and understand screenshots on a computer screen. This feature is achieved through the VNC protocol, allowing the agent to view real-time images of the desktop operating system.
• Action generationBased on observed screenshots, ScreenAgent can generate corresponding mouse and keyboard actions. These actions are output as JSON-formatted command sequences, including moving the mouse, clicking, double-clicking, scrolling, dragging, and keyboard input.
• Task planningScreenAgent can break down complex tasks into a series of subtasks based on user task prompts and plan corresponding action sequences for each subtask. This involves understanding, decomposing, and formulating strategies for the task.
• Execute actionAfter the planning phase, ScreenAgent executes the planned subtasks by sending mouse and keyboard commands to the computer to achieve the user's goals.
• Reflection and EvaluationAfter executing an action, ScreenAgent will evaluate the result and decide whether to retry the current subtask, continue to execute the next subtask, or adjust the entire plan.

ScreenAgent's technical principles

• Visual Language Model (VLM):
  • VLM is a model that combines visual and language processing capabilities, enabling it to understand image content and generate corresponding natural language descriptions.
  • In ScreenAgent, VLM is used to parse screenshots, understand user task prompts, and plan a series of actions to complete the task.
• Enhance the learning environment:
  • ScreenAgent interacts with a real computer screen via the VNC protocol, creating a reinforcement learning environment. In this environment, the agent can observe the screen state (state space), perform actions (action space), and receive rewards (reward function) based on the performance results.
• Control process:
  • PlanningThe agent breaks down the task and plans a series of subtasks and corresponding action sequences based on the current screenshot and task prompts.
  • ExecutionThe intelligent agent manipulates the computer interface by sending mouse and keyboard commands based on the output of the planning phase.
  • ReflectingThe agent evaluates the results of the actions performed and decides whether to retry, continue, or adjust the plan.
• Datasets and Evaluation:
  • The ScreenAgent dataset contains screenshots and action sequences of various everyday computer tasks, used for training and evaluating models.
  • CC-Score (Vision Language Computer Control Score) is a fine-grained evaluation metric used to measure the performance of an agent in computer control tasks.
• Model training:
  • The ScreenAgent model is trained on the ScreenAgent dataset to learn how to effectively plan, execute, and reflect on complex computer control tasks. Various techniques are employed during training, such as supervised learning, reinforcement learning, and human feedback loops (RLHF).