AB
AB
AiBoss

All Categories

project

UI-TARS Desktop - ByteDance's open-source desktop AI GUI intelligent agent tool

UI-TARS Desktop is an open-source desktop automation tool developed by ByteDance, based on a visual language model. It supports controlling computer operations via natural language commands, such as opening files, browsing web pages, and operating software, and can accurately recognize screen input...

What is UI-TARS Desktop?

UI-TARS Desktop is an open-source desktop automation tool developed by ByteDance, based on a visual language model. It supports controlling computer operations via natural language commands, such as opening files, browsing web pages, and operating software, accurately recognizing screen elements and enabling interaction. Compatible with Windows, Linux, and macOS systems, it offers both local and remote operation capabilities, allowing users to choose different scale models based on their hardware. A built-in script editor facilitates user-defined automation tasks. UI-TARS Desktop is open-source under the Apache 2.0 license, free to use, and supports commercial development, making it suitable for various automation scenarios such as data processing and remote maintenance.

Main functions of UI-TARS Desktop

  • Natural Language ControlUsers can directly control their computers using natural language commands, such as "open an application," "search for a specific file," or "fill out a form," without the need for complex programming or manual operation.
  • Visual recognition and interactionIt supports screenshot and visual recognition technology, and can automatically locate interface elements on the screen (such as buttons, input boxes, etc.) and perform precise mouse clicks, keyboard inputs and other operations, making it suitable for complex visual tasks.
  • Cross-platform supportIt is compatible with Windows and macOS systems, and can run on different operating systems to meet diverse needs.
  • Remote controlIt supports remote control of computers and browsers, allowing users to connect to remote devices via the network to perform tasks or assist others in solving problems.
  • Real-time feedbackThe system displays the operation progress and status in real time when executing instructions, allowing users to understand the task execution status at any time.
  • Data securityLocal deployment ensures that operation logs and data are stored on the user's device, guaranteeing privacy and security.

How to use UI-TARS Desktop

  • Download and InstallationVisit the Releases page of the GitHub repository to download the latest version of the UI-TARS Desktop application. For macOS, the downloaded version will be... UI TARS Drag the application to Applications Folder. If the application cannot be opened, you can enter the command in the terminal. sudo xattr -dr com.apple.quarantine /Applications/UI\ TARS.app To repair, Windows users can double-click the downloaded installation file and follow the prompts to install. Homebrew users can also use the command line. brew install ui-tars-desktop Quick installation.
  • Deployment ModelUI-TARS offers two deployment methods: cloud deployment and on-premises deployment (vLLM). For cloud deployment, HuggingFace Inference Endpoints is recommended. For on-premises deployment, you need to install [the necessary software] first. vllm>=0.6.1Then download the 2B, 7B, or 72B model, and finally run the command. python -m vllm.entrypoints.openai.api_server --served-model-name ui-tars --model <path to your model> Start the OpenAI API service and fill in the API information in the settings.
  • Startup and UseOpen the UI-TARS Desktop application, enter a natural language command, such as "open your browser and search for 'AI technology'", and it will automatically parse and execute the corresponding operation, providing real-time feedback on the execution status.
  • Explore and customizeSee the README and contribution guidelines on GitHub for more advanced usage information. Developers can extend functionality through the UI-TARS SDK to create their own custom automation agents.

UI-TARS Desktop project address

  • Github repositoryhttps://github.com/bytedance/UI-TARS-desktop

Application scenarios of UI-TARS Desktop

  • Office AutomationAutomatically process Excel data, such as batch modifying table content and generating charts; quickly open commonly used office software and perform operations, saving manual operation time.
  • Webpage operationAutomatically logs into web pages, fills out forms, and browses web page content, making it suitable for scenarios that require frequent web page operations, such as e-commerce operations and information collection.
  • Software InteractionIt can operate professional software such as Photoshop to automatically perform complex image processing tasks; and control video editing software to complete editing, adding special effects, and other operations.
  • Remote Operations and MaintenanceIt connects to remote computers to perform system maintenance, software updates, troubleshooting, and other operations, making it especially suitable for IT operations and maintenance personnel.
  • Game AssistantAutomatically complete repetitive tasks in the game, such as character leveling and resource collection, to enhance the gaming experience.
  • Learning aidsIt automatically searches for learning materials, organizes notes, and plays instructional videos to help users learn more efficiently.
Back