EvoCUA - Meituan's open-source general-purpose multimodal computer operation model

What is EvoCUA?

EvoCUA (Evolving Computer Use Agent) is an open-source multimodal model from Meituan, focusing on automating computer use tasks. EvoCUA uses natural language commands and screenshots to perform end-to-end multi-turn operations on software such as Chrome, Excel, and PowerPoint, significantly improving computer usage capabilities. In the OSWorld benchmark test, EvoCUA ranked first among open-source models with a task completion rate of 56.7%, surpassing many well-known models. EvoCUA's innovative data synthesis and training methods can significantly improve the computer usage capabilities of open-source visual language models without compromising general performance.

Main functions of EvoCUA

• Multi-turn interactionIt supports multi-round interaction with the desktop environment, and completes the operation step by step according to the task requirements.
• Task automationIt supports completing complex computer tasks based on natural language instructions, such as generating reports in Excel and searching for information in a browser.
• Multimodal inputBy combining visual (screenshots) and linguistic (natural language commands) input, we can more accurately understand task requirements.
• High-efficiency executionBy optimizing algorithms and model structures, tasks can be completed in fewer steps, thus improving efficiency.
• Open source and scalabilityAs an open-source model, EvoCUA allows users to customize and extend it according to their needs.

EvoCUA's technical principles

• Data synthesis and training methods
  EvoCUA employs innovative data synthesis methods to generate large amounts of high-quality training data, simulating real-world computer operation tasks. Its training method significantly improves computer performance without compromising the model's general performance, and through optimized training strategies, the model excels in multimodal tasks.
• Reinforcement learning and optimizationEvoCUA utilizes reinforcement learning techniques during training, optimizing the model's behavior strategy through a reward mechanism. The model continuously tries and learns through multiple rounds of interaction, gradually finding the optimal operational path and achieving efficient task completion in complex tasks.
• Model ArchitectureEvoCUA is built upon Large Language Models (LLMs) and Visual Language Models (VLMs), possessing powerful language understanding and generation capabilities while also processing visual information. Through specific prompts and parsing mechanisms, the model translates natural language instructions into concrete operational commands, enabling automated operation of the desktop environment.