LiveWorld - A generative video world model developed by the University of Adelaide and others.

What is LiveWorld?

LiveWorld is a generative video world model jointly developed by the University of Adelaide, the Australian National University, and other institutions. Its core solution is to address the problem of out-of-view dynamics. When an object leaves the camera's field of view, traditional models freeze its state. LiveWorld, by explicitly decoupling world evolution from observation and rendering, allows dynamic entities to continue to move forward even when they are not visible, achieving a true 4D world simulation.

LiveWorld's main functions

• Dynamic Entity DetectionAutomatically identify moving entities such as people, animals, and vehicles in videos based on Qwen3-VL and SAM3.
• Virtual monitor registrationRegister a Monitor with a fixed perspective for each newly discovered dynamic entity to continuously track its state changes.
• Dynamic evolution outside the field of visionEven after the camera is moved away, the Monitor continues to perform physical actions in the background (such as the dog finishing its food and walking away), rather than freezing the state.
• Static environment accumulationThe Stream3R SLAM framework is used to incrementally fuse static backgrounds into a global 3D point cloud, providing a spatial foundation.
• State-aware renderingThe evolved dynamic 4D point cloud and static 3D point cloud are projected onto the target camera trajectory to generate coherent observation frames.
• Maintaining appearance consistency: Retrieve historical reference frames using Appearance LoRA to ensure that entity identities and textures do not drift in long sequences.

LiveWorld's technical principles

• World state decouplingIt represents the world as a structured approximation of a static 3D background plus dynamic 4D entities, rather than directly predicting 2D frames.
• Evolution-Rendering SeparationThe world first updates its global state through an evolution operator, and then generates observations by combining the camera pose with a rendering operator, thus decoupling the two processes.
• Monitor mechanismDeploy virtual monitors at fixed anchor points and use Evolution Engine to autonomously drive local events.
• SLAM Spatial MemoryIt uses the feedforward SLAM framework Stream3R to fuse static backgrounds in real time, supporting long-term revisiting and perspective changes.
• State injection generationThe projected geometry is injected into the video diffusion model via the State Adapter to constrain the position, structure, and motion of objects.
• Closed-loop production lineThe process involves observing a new area, registering dynamic events, monitoring the background status, and rendering the latest status, creating a continuous loop.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use LiveWorld

• Environmental preparationClone the LiveWorld code repository from GitHub and install the necessary dependencies such as PyTorch, Stream3R, Qwen3-VL, SAM3, and Wan2.1-14B-T2V.
• Input configurationProvide a preceding video frame as the initial observation and define the target camera trajectory and text prompts describing the subsequent behavior of the dynamic entity.
• Dynamic detectionThe system calls Qwen3-VL and SAM3 to automatically scan the preceding frame and identify active dynamic entities such as people, animals, or vehicles.
• Monitor registrationEach newly discovered entity will register a virtual Monitor with a fixed perspective at its location, which will serve as an anchor point for subsequent evolution outside the field of view.
• Evolution outside the field of visionAs the camera moves away along the trajectory, the Monitor continues to use the Evolution Engine in the background to generate subsequent video of the area, advancing the physical movement rather than freezing the state.
• Static memory constructionThe system runs Stream3R SLAM in parallel, merging the incremental background regions observed in history into a global static 3D point cloud.
• State renderingWhen the camera reaches the target position, the system projects the evolved dynamic 4D point cloud and static 3D point cloud onto the target viewpoint, and generates the final observation frame through State Adapter and Appearance LoRA.

LiveWorld's core advantages

• Breaking the static world assumptionThis is the first time that the problem of "outside-field dynamics" has been formalized and solved, breaking the limitation of existing models that only evolve within the field of view.
• Consistency of long-sequence eventsIn the second revisit of the LiveBench benchmark, VQA-Acc reached 54.620, far exceeding competitors such as Spatia.
• Multi-event parallel advancementSupports multiple Monitors simultaneously evolving different events outside the field of view, achieving a Full Succ rate of 26%.
• New perspective geometric consistencyThe dynamic point cloud Chamfer Distance is reduced to 0.135, and the correct spatial position is still maintained when revisiting from a new perspective.
• Modular and scalableThe three modules of static memory, dynamic evolution, and state rendering work together and can be optimized and replaced independently.

LiveWorld's project address

• Project official website: https://zichengduan.github.io/pages/LiveWorld/index.html
• GitHub repository: https://github.com/ZichengDuan/LiveWorld
• HuggingFace model libraryhttps://huggingface.co/ZichengD/LiveWorld
• arXiv technical paper: https://arxiv.org/pdf/2603.07145

Comparison of LiveWorld's similar products

Application scenarios of LiveWorld

• Agent trainingIt provides an evolving, interactive virtual environment for embodied intelligent agents, supporting reasoning about events outside their field of vision.
• Autonomous driving simulationSimulates the dynamic changes of invisible areas (such as blind spots) in traffic scenarios to improve decision-making safety.
• Interactive games: To create an open-world game where NPCs and events continue to progress logically even after the player leaves.
• Synthetic data generationGenerate large-scale training data with long-term temporal consistency and complex event logic.
• Robot navigation planningIt supports robots in maintaining their beliefs about the dynamic state of unobserved areas during the exploration process.