CombatVLA - A VLA model specifically designed for 3D action games, launched by Taotian Group.

What is CombatVLA?

CombatVLA is a high-efficiency visual-language-action (VLA) model developed by the Future Living Lab team at Taotian Group, specifically designed for combat missions in 3D action role-playing games (ARPGs). The model is based on a 3B parameter scale and is trained using video motion pairs collected by a motion tracker, with the data formatted as "Action-Oriented Thinking" (AoT) sequences. The model employs a three-stage progressive learning paradigm, progressing from video-level to frame-level and then to a truncation strategy, to achieve efficient reasoning. CombatVLA outperforms existing models in combat understanding benchmark tests, increasing reasoning speed by 50 times and achieving a higher mission success rate than human players.

Main functions of CombatVLA

• Efficient combat decision-makingCombatVLA can make real-time combat decisions in complex 3D game environments, including dodging attacks, activating skills, and restoring health, with a decision-making speed 50 times faster than traditional models.
• Combat understanding and reasoningCombatVLA can assess the enemy's status, determine the enemy's attack intentions, and deduce the optimal combat actions, demonstrating a significantly better combat understanding than other models.
• Action instruction generationThe model can output specific executable keyboard and mouse operation commands, such as pressing specific keys or performing mouse operations to control game characters to fight.
• Generalization abilityCombatVLA demonstrates strong generalization ability across missions of varying difficulty and in different games, effectively executing combat missions in unfamiliar game scenarios.

CombatVLA Technical Principles

• motion trackerIt collects game operation data from human players based on motion trackers, including keyboard and mouse operations, and simultaneously captures game screens to generate video motion pair data.
• Action-Oriented Thinking (AoT) SequenceThe collected data is transformed into a sequence of "action-of-thought" sequences, with each action accompanied by a detailed explanation, to help the model understand the semantics and logic of the actions.
• Three-stage progressive learning:
  • Phase 1Video-level AoT fine-tuning allows the model to gain an initial understanding of the combat environment.
  • Phase 2Frame-level AoT fine-tuning helps the model understand the strict alignment between actions and preceding frames.
  • Phase 3Frame-level truncation AoT fine-tuning introduces a special marker <TRUNC> to truncate the output to accelerate inference.
• Adaptive Action Weight LossBy using action alignment loss and modal contrast loss, the training process of the model is optimized to ensure that the model accurately outputs key actions.
• Action Execution FrameworkThe action commands output by the model are converted into actual keyboard and mouse operations based on the action execution framework, enabling automatic control of the game character.

CombatVLA project address

• Project official websitehttps://combatvla.github.io/
• GitHub repositoryhttps://github.com/ChenVoid/CombatVLA
• arXiv technical paper: https://arxiv.org/pdf/2503.09527

Application scenarios of CombatVLA

• 3D Action Role-Playing Game (ARPG)Real-time control of game characters in combat enables efficient decision-making and action execution, enhancing the gaming experience.
• Game testing and optimization: Help developers test the game's combat system and identify and optimize potential issues.
• eSports trainingIt provides intelligent opponents for esports players, helping users practice combat skills and tactics to improve their competitive level.
• Game content creationIt assists game developers in generating battle scenes and storylines, and quickly constructing complex game levels and missions.
• Robot controlIn the real world, this technology extends to robot control, enabling robots to make rapid decisions and execute actions in dynamic environments.