Solaris - A multi-user video world generation model open-sourced by Xie Saining's research team

What is Solaris?

Solaris is the first multiplayer video world generation model capable of simultaneously generating consistent first-person perspectives for two players in Minecraft. Breaking away from the limitations of existing models that only support single-player gameplay, it ensures spatial consistency across player perspectives—when one player builds or moves, the other's perspective reflects the changes synchronously. The team developed the SolarisEngine data system, collecting 12.6 million frames of multiplayer game data, and innovatively introduced the Checkpointed Self Forcing training method to solve the memory bottleneck of long sequences.

Solaris main functions

• Multi-person perspective synchronous generationSolaris can generate consistent first-person video for two players simultaneously, ensuring spatial consistency across player perspectives. When one player performs an action, the other player's perspective changes in real time.
• Long-term stable generationThrough Checkpointed Self Forcing technology, Solaris can generate stable video sequences of up to 224 frames (11.2 seconds), effectively avoiding visual degradation caused by error accumulation.
• Action condition controlThe model accepts complete Minecraft action inputs (including movement, camera, digging, placement, etc.), and the generated video strictly follows the given action sequence.
• Complex dynamic simulationSolaris can simulate complex game dynamics such as backpack state synchronization, weather changes, physical construction and destruction, and PvP combat.

Solaris Technical Principles

• Multi-person DiT architectureBased on the MatrixGame 2.0 single-player diffusion Transformer, it supports full Minecraft input by expanding the action space, introduces a cross-player self-attention layer to realize information exchange between two players, adds player ID embedding to distinguish different perspectives, and keeps the other modules (cross-attention, FFN) unchanged in single-player settings.
• Four-stage progressive trainingStarting with pre-trained weights for single-player mode, we fine-tuned the weights on the VPT dataset to fit the Minecraft action space. Then, we switched to multiplayer data to train a bidirectional model as the teacher, causally transformed into a sliding window generator, and achieved stable generation of long sequences through Checkpointed Self Forcing.
• Checkpointed Self ForcingTo address the memory bottleneck of sliding window autoregression, this method first generates and caches clean frames and noisy states without gradients. Then, it performs a single-pass parallel recalculation using a custom attention mask to strictly reproduce the sliding window dependencies, thus freeing up memory.O(Lt ⋅Ls ) Down toO(Lt ) It also supports KV caching gradient backhaul to improve generation quality.
• Solaris Engine Data SystemTo address the lack of multiplayer support in existing frameworks, the team built a separate architecture for the controller and camera in the official Minecraft client based on Mineflayer. The status was synchronized in real time through server plugins, and parallel scaling and automatic fault recovery were achieved through Docker containerization. Ultimately, 12.6 million frames of motion-annotated multiplayer game data were collected.

Solaris project address

• Project official websitehttps://solaris-wm.github.io/
• GitHub repositoryhttps://github.com/solaris-wm/solaris
• HuggingFace model libraryhttps://huggingface.co/collections/nyu-visionx/solaris-models
• arXiv technical paper: https://arxiv.org/pdf/2602.22208

Solaris application scenarios

• Embodied Intelligence Training and AssessmentAs a multi-agent world simulator, it provides synthetic training data for robots and game AI, supports policy learning, inference-time planning and safety assessment, and avoids the high cost of trial and error in real-world environments.
• Multi-agent collaborative researchSimulate multi-person collaborative tasks (such as joint construction and team combat) to train the collaboration and communication capabilities of AI agents, and to study emergent behavior and social intelligence.
• Visual-Language-Motion Model DevelopmentThe model can generate large-scale, multi-view video-action-language aligned data, supporting pre-training and fine-tuning of VLA models, thus compensating for the scarcity of real human multi-person interaction data.
• 3D Scene Understanding and Spatial Reasoning BenchmarkAs a controllable testing platform, it evaluates the model's performance in core 3D understanding capabilities such as viewpoint consistency, object persistence, and spatial memory.