Matrix -Game 3.5 - A real-time streaming interactive world model open-sourced by Kunlun Wanwei.

What is Matrix-Game 3.5?

Matrix-Game 3.5 is an open-source real-time streaming interactive world model from the Kunlun Wanwei Riemann Power team. It utilizes Patch Memory and Warped ProPE to achieve 3D spatial-level long-term memory and geometrically consistent modeling. The model supports 720P/20FPS single-card real-time generation, has minute-level scene backtracking capabilities, and responds to real-time keyboard and mouse input. Moving from a game engine to a general-purpose physical world simulation, it provides open-source infrastructure for robot training, autonomous driving simulation, and embodied intelligence.

Main features of Matrix-Game 3.5

• Patch Memory (Long-term Memory)The system divides historical footage into image patches in three-dimensional spatial coordinates, retrieves and reassembles them according to their spatial location, and solves the problems of object reproduction and scene drift. The object reproduction score breaks through the previous industry limit of 0.6.
• PRoPE + Warped RoPE Geometric EncodingIntegrating camera projection matrix encoding into the Transformer spatiotemporal position representation enables the model to perceive the relationship between camera rotation, translation, and projection, achieving precise viewpoint control and geometric consistency.
• Dynamic-static decoupling memoryStatic scenes are maintained by Patch Memory, while dynamic subjects maintain identity consistency through lightweight Reference Tokens to avoid moving objects polluting the memory and causing ghosting.
• Real-time streaming generationBy compressing the sampling steps to 3 steps through distillation, and combining block causal reasoning, KV Cache, INT8 quantization and VAE structured pruning, real-time generation of 720P/20FPS on a single GPU is achieved.
• Automated data pipelineWe built three data systems: Unreal Engine, AAA game pipeline, and internet video pipeline, producing over 5 million high-quality video clips and over 10,000 hours of training data, covering over 1,200 game scenarios.
• NPC Interaction and Open World GeneralizationIt supports text-driven world generation, character action control, and multi-agent collaboration.

Technical principles of Matrix-Game 3.5

• Unified Geometric Perceptual Memory FrameworkPRoPE encodes camera pose as a relative position code without introducing additional learnable parameters; Warped RoPE recalculates the spatiotemporal coordinates of historical patches in the current viewpoint, enabling coordinate alignment when reusing memories.
• Progressive distillationThe bidirectional DiT is distilled into a causal generator in two stages: high-quality few-step causal initialization is obtained using Perceptual Flow Matching, and training is performed on the autoregressive trajectory using Self-Rollout DMD (Distribution Matching Distillation). Conditional courses are used to gradually distill the classifier-free, camera-controlled, and memory-based conditional generation.
• Lightweight pluggable architectureThe core interactive functions (PRoPE, Patch Memory, and static/dynamic decoupling) do not require any new parameters and are adapted to different video base models using modular plugins.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Matrix - Game 3.5

• Download ModelObtain the weights and inference code via GitHub (https://github.com/Riemann-Dynamics/Matrix-Game-3.5) or HuggingFace.
• Deployment environmentPrepare a single GPU that supports CUDA, install the dependencies, and then load the model.
• Input controlUse the WASD keys to move, the mouse to control the viewpoint, or input text prompts/camera trajectories to generate an interactive world.
• Real-time interactionThe model streams subsequent frames at 20 FPS, supporting minute-level continuous exploration without losing scene consistency.

The core advantages of Matrix - Game 3.5

• Long-term consistencyFor the first time, a systematic solution has been developed to address the minute-level memory bottleneck of the world model in an open-source solution, ensuring that scene elements remain completely consistent during camera revisits.
• Real-time interactiveUpgraded from an offline video generator to a playable, live world, supporting real-time keyboard/mouse control with latency as low as playable levels.
• Open source ecosystemThe core architecture is completely open source, available on GitHub and HuggingFace, and has been cited as a benchmark by DiT's author Sai-Ning Xie's team (Solaris), NVIDIA, Adobe, and others.
• Zero-parameter incremental designIt does not stack parameters or disrupt the original video distribution, preserving the basic model's native open content generation capabilities.

The project address for Matrix - Game 3.5

• Project official website：https://matrix-game-v3-5.github.io/
• GitHub repository：https://github.com/Riemann-Dynamics/Matrix-Game-3.5
• HuggingFace model library：https://huggingface.co/RiemannDynamics/Matrix-Game-3.5-Base
• Technical Papers：https://matrix-game-v3-5.github.io/paper/Matrix-Game-3.5.pdf

Comparison of Matrix-Game 3.5 with similar competing products

Application scenarios of Matrix-Game 3.5

• AI game engineIt replaces traditional game engines to generate interactive open worlds in real time, reducing the development cost of AAA games.
• Robot Virtual TrainingAs a virtual training sandbox for humanoid robots/robotic arms, it supports low-cost, massive-scale scenario trial and error and long-term task planning.
• Autonomous driving simulation: Construct a digital twin of a traffic scenario based on real physical rules for end-to-end driving strategy training.
• Embodied Intelligence ResearchProvides an interactive physical world simulation environment for embodied agents, supporting joint action-state training.
• XR/Metaverse Content GenerationIt generates immersive virtual spaces in real time, allowing users to freely explore and create content.