Matrix-Game - Kunlun Tech's open-source industrial-grade spatial intelligence model
Matrix-Game is Kunlun Wanwei's open-source industrial-grade 10B+ spatial intelligence model, and an interactive video generation model within the Matrix-Zero world model. The model is based on a two-stage training strategy, generating continuous video based on user input...
What is Matrix-Game?
Matrix-Game is Kunlun Wanwei's open-source industrial-grade 10B+ spatial intelligence model, and an interactive video generation model within the Matrix-Zero world model. Based on a two-stage training strategy, the model generates coherent and controllable interactive videos according to user input. It boasts advantages such as fine-grained user interaction control, high-fidelity visual and physical consistency, and multi-scene generalization capabilities. It can be used in virtual game world construction, film and metaverse content production, and other fields, setting a new benchmark for building a universal virtual world foundation.
The main functions of Matrix-Game
- Controllable video generationUsers can freely explore, manipulate, and even create a virtual world with rich details and reasonable physical rules based on simple keyboard commands and mouse movements.
- Multi-scenario generalizationIt has the ability to generalize to various Minecraft game scenes (such as forests, beaches, deserts, glaciers, etc.) and has the potential to generalize to non-Minecraft game environments.
- Autoregressive long video generationIt supports autoregressive long video generation, achieving a smooth transition between action and perspective, ensuring time consistency and environmental adaptability.
- Systematic assessmentThe proposed unified GameWorld Score standard comprehensively quantifies model performance across four dimensions: visual quality, temporal quality, motion controllability, and understanding of physical rules.
The technical principles of Matrix-Game
- Two-stage training strategyUsing large-scale unlabeled Minecraft game video data, the model is pre-trained to learn the basic features and dynamic patterns of the environment. Fine-grained, controllable training is then performed using controllable Minecraft and Unreal Engine video data with keyboard and mouse control signals, allowing the model to generate corresponding interactive videos based on user input.
- Image to World ModelingA single reference image serves as the starting point for generating interactive videos, without relying on language prompts, and models spatial geometry, object motion, and their physical interactions based on visual signals.
- Autoregressive video generationIt supports an autoregressive approach to extend the generation length, using the last few frames of the previous video segment as the motion context for each generation, progressively generating segments to ensure temporal continuity. During training, random perturbations, random deletions, and classifier-free guidance strategies are introduced to mitigate temporal drift and error accumulation.
- Controllable Interaction DesignKeyboard actions are represented by discrete tokens, while view movement actions are represented by continuous tokens. The control module is based on GameFactory and incorporates a multimodal Diffusion Transformer architecture. A classifier-free guidance strategy is used to improve the robustness of the response to control signals.
The project address for Matrix-Game
- Project official website:https://matrix-game-homepage.github.io/
- GitHub repository:https://github.com/SkyworkAI/Matrix-Game
- HuggingFace model library:https://huggingface.co/Skywork/Matrix-Game
- Technical Papers:https://github.com/SkyworkAI/Matrix-Game
Application scenarios of Matrix-Game
- Virtual game developmentIt can quickly generate diverse game maps and dynamic interactive environments, improving development efficiency and player immersion.
- Film and MetaverseGenerates high-fidelity dynamic scenes, supports immersive experience development, and helps generate creative content quickly.
- Embodied Intelligence TrainingIt provides diverse virtual environments to enhance the training data of embodied intelligent agents and improve their task execution capabilities.
- Education and TrainingTo create virtual teaching and vocational skills training environments to help students and learners better understand and practice.
- Creative content generationIt provides a wealth of materials for creative video production and virtual scene design, supporting the rapid realization of creative ideas.