MineWorld - A real-time interactive world model open-sourced by Microsoft Research.

What is MineWorld?

MineWorld is an open-source real-time interactive world model based on Minecraft, developed by Microsoft Research. It utilizes a vision-action autoregressive Transformer architecture, transforming game scenes and actions into discrete token IDs, which are then trained using predictions of the next token. The model employs a parallel decoding algorithm, achieving a generation speed of 4 to 7 frames per second and supporting real-time interaction. MineWorld outperforms existing models, such as Oasis, in terms of video quality, controllability, and inference speed.

MineWorld's main functions

• High quality of productionBased on the vision-motion autoregressive Transformer, MineWorld can generate coherent, high-fidelity game frames based on vision and motion.
• Strong controllabilityThe model demonstrates accurate and consistent behavior through benchmark tests of motion-following capabilities, and can generate accurate game scenes based on input actions.
• Fast reasoning speedIt employs a parallel decoding algorithm, enabling the model to generate images at a rate of 4 to 7 frames per second, supporting real-time interaction.
• As a game distributorMineWorld predicts game states and actions simultaneously during training and can act as an independent game agent to play the game autonomously.
• Real-time interactive capabilitiesUsers can interact with the model in real time through a web demo or local run, selecting an initial frame, controlling camera movement, and performing game actions.

MineWorld's Technical Principles

• Visual-Motion Autoregressive TransformerMineWorld achieves joint modeling of visuals and motion by transforming game scenes and player actions into discrete token sequences. Specifically:
  • Visual TokenizerThe VQ-VAE architecture is used to segment game scenes into discrete visual labels. The labeler starts from pre-trained checkpoints and is fine-tuned on the Minecraft dataset to achieve high-quality image reconstruction.
  • Action TokenizerThe system quantifies a player's continuous actions (such as mouse movement) into discrete tokens, and categorizes discrete actions (such as moving forward or attacking) into different categories, each represented by a unique token.
  • Transformer decoderEmploying an LLaMA architecture, it receives interleaved sequences of visual and action tags as input and trains by predicting the next tag. The decoder can simultaneously learn rich representations of the game state and the conditional relationships between states and actions.
• Parallel decoding algorithmTo achieve real-time interaction, MineWorld developed a parallel decoding algorithm. This algorithm leverages the spatial dependencies between adjacent image markers while simultaneously predicting spatially redundant markers in each frame. Compared to traditional autoregressive decoding algorithms, it significantly improves generation speed, enabling the model to achieve generation rates of 4 to 7 frames per second at various scales.
• trainThe model is trained by predicting the next label, learning the dynamic evolution of game states and the relationship between actions and states.
• reasoningDuring the inference phase, the model generates subsequent game scenes based on the current game state and actions input. The application of parallel decoding algorithms enables the model to quickly generate high-quality game frames.
• Evaluation indicatorsMineWorld proposes new evaluation metrics to assess the visual quality and motion-following ability of generated scenes. For example, it quantifies the controllability of the model by comparing the accuracy between predicted actions in the generated scene and the real input actions.

MineWorld's project address

• Github repository:https://github.com/microsoft/MineWorld
• HuggingFace model library:https://huggingface.co/microsoft/mineworld
• arXiv technical paper:https://arxiv.org/pdf/2504.08388

Application scenarios of MineWorld

• Embodied Intelligence ResearchMineWorld provides a high-fidelity, interactive virtual environment that simulates complex physical rules and dynamic scenes, making it ideal for research on embodied intelligence. Researchers can use models to train agents to learn how to perform tasks in the virtual environment, such as object localization and navigation, and environmental exploration.
• Reinforcement learning trainingMineWorld's real-time interactivity and high-quality generation make it an ideal platform for reinforcement learning training. Researchers can use the model to quickly generate large amounts of training data, helping agents learn optimal policies in simulated environments.
• Game agency developmentBecause MineWorld predicts both game states and actions during training, it has the potential to act as a game agent. Given an initial game state and actions, the model can iteratively generate future states and actions, simulating a long-term game process.
• Real-time interactive simulationMineWorld's fast reasoning speed (4 to 7 frames per second) enables real-time interaction with game players.
• Video generation and editingMineWorld can generate high-quality, consistent game videos. It can be used for video content creation, such as generating game trailers and tutorial videos.