WorldMem - A world generative model jointly launched by Nanyang Technological University, Peking University, and Shanghai AI Lab.

What is WorldMem?

WorldMem is an innovative AI world generation model developed by Nanyang Technological University, Peking University, and the Shanghai AI Lab. The model addresses the critical issue of lack of consistency in traditional world generation models over long time by introducing a memory mechanism. In WorldMem, agents freely explore diverse scenes, and the generated world maintains geometric consistency despite changes in viewpoint and position. WorldMem supports temporal consistency modeling, simulating dynamic changes (such as the impact of objects on the environment). The model is trained on a large scale on the Minecraft dataset and its effectiveness is validated in real-world scenarios. WorldMem provides a new technological path for building realistic, persistent, and interactive virtual worlds.

Main functions of WorldMem

• Maintain consistencyTo keep the virtual world consistent over a long period of time.
• Simulate dynamic changesSimulate changes that occur over time, such as the impact of objects on the environment (e.g., light melting snow).
• Supports interactionUsers place objects or perform actions in the virtual world, and these interactions are recorded and affect subsequent generation.
• Diverse scene generationSupports free exploration in various virtual scenes (such as plains, deserts, ice fields, etc.).
• Applicable to real-world scenarios: The ability to validate the consistency of generation on real-world datasets.

The technical principles of WorldMem

• Condition generation moduleBuilt upon the Conditional Diffusion Transformer and combined with the Diffusion Forcing training strategy, it supports autoregressive long-term generation. External action signals (such as movement, viewpoint control, and object placement) guide the generation of first-person perspectives.
• Memory Read/Write ModuleThe memory bank stores key historical information from the generation process. Each memory unit contains an image frame and its corresponding state (such as viewpoint pose and timestamp). Memory retrieval uses a greedy matching algorithm, which calculates similarity based on viewpoint overlap and temporal differences to efficiently select the memory units most relevant to the current scene.
• Memory fusion moduleThe current frame's state embedding (pose + time) is combined with that of the memory frame. Attention-based computation is used to extract the memory information most relevant to the current scene, generating fused features to guide the generation of the current frame. Pose is represented using Plücker coordinates, and timestamps are mapped based on MLP. A relative embedding mechanism is introduced to improve the model's spatial understanding and detail preservation capabilities.

WorldMem project address

• Project official website:https://xizaoqu.github.io/worldmem/
• GitHub repository:https://github.com/xizaoqu/WorldMem
• arXiv technical paper:https://arxiv.org/pdf/2504.12369
• Experience the demo online:https://huggingface.co/spaces/yslan/worldmem

Application scenarios of WorldMem

• Virtual GamesGenerate a consistent virtual game world that supports free exploration and environmental interaction.
• VR/ARCreate persistent and dynamically changing virtual environments to enhance immersion.
• autonomous drivingSimulates real-world traffic scenarios for use in testing autonomous driving systems.
• Architectural DesignGenerate virtual building environments to assist in the evaluation of design schemes.
• educateCreate an interactive learning environment to support students in conducting experiments and explorations.