Lumine - ByteDance's 3D open-world general-purpose AI agent

What is Lumine?

Lumine is a general-purpose AI agent launched by ByteDance, enabling real-time perception, reasoning, and action in 3D open-world games. Based on the Qwen2-VL-7B-Base model, Lumine integrates perception, thinking, and action through a human-like interaction paradigm. Lumine can perceive the game environment in real time and perform complex tasks, including combat, puzzle-solving, NPC interaction, and GUI operations. Lumine performed exceptionally well in Genshin Impact, completing main quests lasting several hours, and demonstrated cross-game generalization capabilities in other games. Through multi-stage training, Lumine gradually improves its autonomous thinking and operational abilities, providing a new direction for the future development of embodied intelligence.

Lumine's main functions

• Task executionIt can autonomously complete complex, long-term tasks, such as completing the main storyline in Genshin Impact, which can last for several hours.
• Combat abilityDynamically track enemies, shoot accurately, switch characters to perform combo attacks, and efficiently open treasure chests.
• Puzzle solving abilitySupports solving various puzzle challenges, such as collecting Anemoculus and activating Elemental Monuments.
• NPC Interaction: Able to have stable conversations with NPCs and complete tasks.
• GUI operationPerform 2D interface operations like human players, such as crafting items and using teleportation anchors.
• Cross-game generalizationNo additional tweaks are needed; it can complete missions in other games (such as Honkai Impact 3rd and The Tide).

Lumine's technical principles

• Perceived SpaceIt processes one frame of game footage every 200ms, preserves historical reasoning patterns, and provides contextual information for decision-making.
• Hybrid thinking strategiesIt performs reasoning in critical scenarios (such as sudden environmental changes or plan failures) and directly outputs actions in simple scenarios, improving efficiency.
• Keyboard and mouse operation modelingDefine all operations as mouse movement and key press sequences, and perform three-stage training using high-quality data:
  • Pre-trainingLearn basic visual-motor skills.
  • Command Follow-up Training: Associate language commands and actions.
  • Decision-making reasoning trainingLearn to plan and revise independently to complete long-term tasks.
• Real-time optimizationBy managing context and optimizing across multiple dimensions, latency is reduced and real-time interaction is ensured.

Lumine's project address

• Project official websitehttps://www.lumine-ai.org/
• arXiv technical paper: https://arxiv.org/pdf/2511.08892

Application scenarios of Lumine

• Game Development and TestingUsed in game automation testing, it helps developers quickly discover vulnerabilities, performance issues, and user experience problems, while also assisting in generating intelligent NPC behaviors and task designs, thereby improving game development efficiency.
• Games and EntertainmentAs an AI teammate or opponent, it provides players with a more challenging and fun interactive experience, or assists players in completing complex tasks in single-player games, thereby improving game completion.
• Education and TrainingUsed in virtual training environments to provide task training for students or professionals, or to assist learning in educational games, helping students master knowledge and skills through tasks and challenges.
• Virtual World and MetaverseAs virtual characters, they interact with users in the metaverse, providing companionship or services, helping users generate tasks, storylines, or interactive content in the virtual world, and enriching the user experience.
• Industry and manufacturingIt enables tasks and process optimization in industrial simulation environments, helps design efficient work patterns, and is used in the training of intelligent robots to improve their autonomous decision-making and operational capabilities.