Vid2World - Tsinghua University and Chongqing University jointly launch a framework for converting video models into world models.

What is Vid2World?

Vid2World, an innovative framework jointly developed by Tsinghua University and Chongqing University, supports the transformation of full-sequence, non-causal passive video diffusion models (VDM) into autoregressive, interactive, and action-conditional world models. Based on two core technologies—video diffusion causation and causal action guidance—the model addresses the shortcomings of traditional VDM in causal generation and action conditionation. Vid2World performs exceptionally well in complex environments such as robot manipulation and game simulation, supporting the generation of high-fidelity, dynamically consistent video sequences and action-based interactive predictions. Vid2World opens up new avenues for improving the practicality and prediction accuracy of world models and has broad application prospects.

Vid2World's main functions

• High-fidelity video generationGenerate predictions that are highly similar to real videos in terms of visual fidelity and dynamic consistency.
• Action ConditioningIt generates corresponding video frames based on the input action sequence, supporting fine-grained motion control.
• Autoregressive generationThe video is generated frame by frame using an autoregressive method, with each generation step relying only on past frames and actions.
• Causal reasoningThe model can perform causal inferences, and its predictions rely solely on past information and are not affected by future information.
• Support downstream tasksSupports interactive tasks such as assisting robot operation and game simulation.

The technical principles of Vid2World

• Causation of video spreadVideo Diffusion Model (VDM) denoises the entire video sequence simultaneously. However, this full-sequence generation method is unsuitable for causal inference because future frames influence past frames. To achieve causal generation, Vid2World modifies the pre-trained VDM. The temporal attention layer applies a causal mask, restricting the attention mechanism to accessing only past frames, thus achieving causality. The temporal convolutional layer introduces a hybrid weight transfer scheme, preserving pre-trained weights while adapting the model to causal convolutional layers. Based on Diffusion Forcing, noise levels are independently sampled for each frame during training, allowing the model to learn combinations of noise levels between different frames, supporting autoregressive generation.
• Causal action guidanceTo enable the model to respond to fine-grained actions, Vid2World introduces a causal action guidance mechanism. Each action is encoded using a lightweight multilayer perceptron (MLP) and added to the corresponding frame. During training, each action is independently dropped with a fixed probability, forcing the model to learn both conditional and unconditional score functions. During testing, the responsiveness to action changes is adjusted based on a linear combination of the conditional and unconditional score functions. By independently dropping actions, the model learns the impact of actions on the generated results, better responding to action inputs during autoregressive generation.

Vid2World project address

• Project official website:https://knightnemo.github.io/vid2world/
• HuggingFace model library:https://huggingface.co/papers/2505.14357
• arXiv technical paper:https://arxiv.org/pdf/2505.14357

Application scenarios of Vid2World

• robot operationGenerate high-fidelity predictions to assist robot task planning.
• Game SimulationGenerate videos that closely resemble real-world gameplay, aiding in the development of neural game engines.
• Strategy EvaluationSimulate the execution results of different strategies to help optimize strategies.
• Video predictionPredicting subsequent frames based on existing frames and action sequences, used in video completion, etc.
• Virtual environment constructionGenerate virtual scenes that respond to actions, enhancing the interactivity of virtual reality.