Matrix-Game 3.0 - A real-time interactive world model launched by Kunlun Wanwei

What is Matrix-Game 3.0?

Matrix-Game 3.0 is a real-time interactive world model launched by Skywork AI, supporting real-time generation at 720p@40FPS. The model employs an error buffering mechanism for self-correction, combined with camera-based sensory memory retrieval to ensure long-term consistency, enabling stable generation of interactive videos lasting several minutes. Matrix-Game 3.0 is trained using Unreal Engine synthetic data, AAA game data, and real video. The 5B distillation version supports 3-step fast sampling, while the 28B MoE version further improves quality. Serving as the foundation for the "Cat Forest Academy 2.0" platform, it aims to create an interactive AAA-level AI game world.

Main features of Matrix-Game 3.0

• Real-time interactive generationSupports keyboard/mouse input and generates interactive 720p resolution video in real time at 40FPS.
• Long-term memory retentionMaintaining spatiotemporal consistency of scenes and objects in minute-long videos through camera-sensory memory retrieval.
• Self-correction capabilityThe model uses an error buffering mechanism to automatically correct prediction biases, reducing image drift and accumulated errors.
• Multimodal controlSupports combined conditional control of text prompts, motion input, and camera pose.
• Efficient inference deploymentThe model supports 5B distillation model which can run in real time with only 3 sampling steps, and the 28B MoE version provides higher generation quality.

The technical principles of Matrix-Game 3.0

• Data EngineIt integrates Unreal Engine synthetic data, large-scale AAA game automated data acquisition, and real-world video enhancement to build an industrial-grade infinite data pipeline, producing Video-Pose-Action-Prompt quadruple training data.
• Error-aware basic modelThe model employs a bidirectional Diffusion Transformer to jointly model historical frames, noisy current frames, and action conditions. An error collection mechanism records the prediction residuals and injects them into the training, enabling the model to learn self-correction and bridge the gap between training and inference.
• Memory enhancement generation: Introduce camera-aware memory frame retrieval on the basic model, select relevant historical content based on camera pose and field of view overlap, and achieve long-range spatiotemporal consistency through a joint self-attention mechanism and Plücker geometric coding.
• Few-step distillation for training inference alignmentBased on the multi-stage autoregressive distillation strategy of Distribution Matching Distillation, the student model performs multi-stage rollout to simulate actual inference, ensuring that the training and inference distributions are consistent. Real-time inference is achieved by combining INT8 quantization and VAE decoder distillation.

Key information and usage requirements for Matrix-Game 3.0

• Release timeMarch 27, 2026 (2026 Zhongguancun Forum)
• Development TeamKunlun Tech / Skywork AI
• Model localizationReal-time streaming interactive world model
• Core Competencies720p@40FPS real-time generation, minute-level long-range memory consistency
• Model versionOffers 5B basic version / 5B distillation version / 28B MoE version
• Official resourcesGitHub / HuggingFace / Demo Site
• Hardware Requirements
  • GPUNVIDIA A/H series (supports single-card or multi-card inference)
  • systemLinux
  • Memory64GB RAM

The core advantages of Matrix-Game 3.0

• Industrial-grade real-time performanceThe 5B distillation model can achieve real-time generation at 720p@40FPS, breaking through the speed bottleneck of interactive video generation and reaching a practically deployable industrial standard.
• Long-range consistency breakthroughIt pioneered an error buffer and camera-sensory memory retrieval mechanism to solve the drift problem in long video generation using diffusion models, supporting stable output at the minute level.
• Training inference alignmentThe multi-stage autoregressive distillation strategy ensures consistency between training and inference distributions, and achieves the quality of a 50-step basic model in just 3 sampling steps, improving efficiency by 16 times.
• Leading data engineIntegrating Unreal Engine compositing, AAA game capture, and realistic video enhancement to build an infinitely high-quality data pipeline.

How to use Matrix-Game 3.0

• Environmental preparationConfigure a conda environment for Python 3.12 on a Linux system and install the FlashAttention dependency library.
• Warehouse installationClone the GitHub repository to your local machine and use pip to install all dependencies listed in requirements.txt.
• Model DownloadDownload the pre-trained model weight file from HuggingFace using the huggingface-cli command.
• Input preparationPrepare a starting image and text prompts describing the scene as generation conditions.
• Basic ReasoningRun the torchrun command to start the generation process, and set the resolution to 704×1280, INT8 quantization, and 3-step sampling to achieve real-time generation.
• Parameter adjustmentChoose the basic model (50 high-quality steps) or the distillation model (3 quick steps) according to your needs, and enable interactive mode to customize action input.

Project address for Matrix-Game 3.0

• Project official websitehttps://matrix-game-v3.github.io/
• GitHub repository: https://github.com/SkyworkAI/Matrix-Game/tree/main/Matrix-Game-3
• HuggingFace model libraryhttps://huggingface.co/Skywork/Matrix-Game-3.0
• Technical Papers: https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-3/assets/pdf/report.pdf

Analysis of competing products similar to Matrix-Game 3.0

Application scenarios of Matrix-Game 3.0

• AI game developmentAs the foundation of the "Cat Forest Academy 2.0" platform, it constructs an interactive 3A-level AI game world, realizing real-time dynamic scene generation and player action response.
• Virtual World ConstructionThe model can create coherent open-world environments that run for minutes, supporting free exploration from first-person/third-person perspectives, and is suitable for metaverse and virtual simulation.
• Real-time interactive entertainmentIt supports real-time keyboard and mouse control, generates smooth interactive video content, and is applied to AI-driven interactive storytelling and immersive experiences.
• Game Prototype DesignIt can quickly generate diverse game scenes and dynamic effects, accelerating concept verification and visual preview in the early stages of game development.