Multiverse - Enigma Labs' world's first AI-generated multiplayer game model

What is Multiverse?

Multiverse, developed by the Israeli team Enigma Labs, is the world's first AI-generated multiplayer game model. It's a multiplayer racing game where players can overtake, drift, and accelerate, each action impacting and reshaping the game world in real time. The model uses AI technology to generate game visuals in real-time, ensuring that both players see the same logically consistent world. Based on a diffusion model, it fuses player perspectives and actions to generate a coherent and consistent game visual. Multiverse's core technology lies in its innovative multiplayer world model architecture, which solves the challenge of consistent perspectives in multiplayer games through joint action vectors and dual-viewpoint stacking technology. The model's training cost is only $1500 and it can run on a standard PC. The project's code, data, weights, architecture, and research results are fully open-source, providing new possibilities for the application of AI in multiplayer games.

The main functions of Multiverse

• Real-time multi-user interactionIt supports real-time interaction between two players in the same virtual world, such as overtaking and collisions in racing games, with both players maintaining the same perspective.
• Dynamic world generationThe game screen is generated in real time based on the player's actions and operations.
• High-efficiency frame predictionAccurately predict future game frames to ensure smooth and consistent gameplay.
• Low-cost operationIt supports running on ordinary personal computers, without the need for high-end hardware, thus lowering the barrier to entry.

The technical principle of Multiverse

• Multiplayer game architectureTo build a multiplayer game world model, the core building blocks mentioned above were retained, but the structure was deconstructed—the inputs and outputs were reconnected, and the training process was redesigned from scratch to achieve true cooperative gameplay.
  • Action embedder: Get the actions of the two players and output an embedding representing them.
  • Denoising NetworkA diffusion network that can simultaneously generate frames for two players as a single entity, based on the frames and motion embeddings between the two players.
  • upsamplingThe upsampler here receives frames from both players and calculates the upsampled version.
• Perspective merging solutionTo create a multiplayer gaming experience, the model needs to collect frames and actions from both players and output their respective predicted frames. The key is that these two outputs cannot just look aesthetically pleasing; they need to be internally consistent. Multiverse proposes a workaround: stitching the perspectives of the two players into a single image, fusing their inputs into a joint action vector, and treating it all as a unified scene. Specifically, this is done by stacking frames along the channel axis, treating the two frames as images with twice the number of color channels. Because the diffusion model here is a U-shaped network, mainly composed of convolutional and deconvolutional layers, the first layer only processes nearby pixels. If the two frames are stacked vertically, then frames are not processed until the middle layer, reducing the model's ability to produce a consistent structure across frames. However, if the frames are stacked along the channel axis, each layer of the network processes the views of both players simultaneously.
• Training methods
  • Context extensionTo accurately predict the next frame, the model needs to receive player actions (such as steering input) and a sufficient number of frames to calculate the speeds of the two cars relative to the road and to each other. Research found that 8 frames (30 frames per second) are sufficient for the model to learn vehicle kinematics, such as acceleration, braking, and steering. However, the relative speed of the two cars is much slower than that of the road. To capture this relative motion, the context size needs to be increased by nearly three times. But doing so would make the model too slow for real-time gameplay, increase memory usage, and significantly reduce training speed. To maintain the context size while providing more temporal information, the authors provided the model with sparse sampling of the previous few frames and actions. Specifically, they provided the model with the most recent 4 frames, and then every 4 frames in the next 4 frames. The earliest frame in the context is 20 frames, or 0.666 seconds ago, which is sufficient to capture the relative motion of the vehicles. This also allows the model to better capture speed and acceleration relative to the road surface, resulting in more dynamic driving effects.
  • Multiplayer trainingTo enable the model to learn driving techniques and interactions in multiplayer games, it needs to be trained in these interactive scenarios. Walking, driving, and other common tasks in the world model typically require only short prediction spans, such as predicting the situation 0.25 seconds into the future. Interactions in multiplayer games require much longer time spans. Within a quarter of a second, the relative motion between players is almost negligible. To train a multiplayer game world model, a longer prediction span is needed. Multiverse trains the model to make autoregressive predictions (at 30 frames per second) up to 15 seconds into the future. To enable such long-term predictions, Multiverse uses a learning-by-course approach, increasing the prediction time from 0.25 seconds to 15 seconds during training. The initial training phase allows for efficient model training, where the model learns low-level features such as car and track geometry. As the model learns to generate coherent frames and model vehicle kinematics, it is trained on higher-level concepts such as player behavior. Increasing the prediction span significantly improves the model's "physical consistency" and inter-frame consistency.
  • Highly efficient long field of vision trainingTraining a model for more than 100 frames presents challenges to VRAM. Loading these frames into GPU memory for autoregressive prediction becomes impractical with larger batches. To address this memory constraint, a paging approach is used for autoregressive prediction. At the start of training, the first batch of data is loaded and used for prediction. Then, the next page of data is loaded, discarding frames that are outside the context window.

Multiverse project address

• Project official website:https://enigma-labs.io/blog
• GitHub repository:https://github.com/EnigmaLabsAI/multiverse
• HuggingFace model library:https://huggingface.co/Enigma-AI

Multiverse dataset

• Data sourceThe data for the team's training model was collected from Sony's game Gran Turismo 4.
• Data collection methodsBased on the in-game replay system, each match is replayed twice, recorded from each player's perspective. The two recordings are then synchronized, aligned with the original two-player match, and merged into a single video showcasing both players simultaneously. Computer vision is used to extract the accelerator, brake, and direction bars displayed on the game screen frame by frame, and then the control commands are deduced. In other words, the controls can be reconstructed entirely from the on-screen information, without the need for additional log files.
• Automatic data generationWrite scripts to send random input to the game's B-Spec mode, automatically triggering matches, recording replay footage from two perspectives, and capturing third-person videos of AI-driven matches.

Application scenarios of multiverse

• Multiplayer game developmentUsed in developing multiplayer online games to provide a more realistic and richer interactive experience.
• VR/AR applicationsCreate a shared virtual environment for multiple users to enhance immersion and social interaction.
• AI Training and ResearchAs an open-source model, it trains intelligent AI agents and studies decision-making and collaboration in complex environments.
• Education and TrainingCreate virtual training scenarios for use in driving, military exercises, or teamwork training.
• Entertainment and SocialDevelop applications such as virtual gatherings and online events to provide novel social experiences.
• Simulation management gameIn simulation management games, players need to perform operations such as resource management, building planning, and city development. Every decision can affect the economic and ecological balance of the entire simulated world.