VerseCrafter - A dynamic, realistic video world model open-sourced by Fudan University and Tencent.

What is VerseCrafter?

VerseCrafter is a dynamic, realistic video world model developed by Fudan University and Tencent PCG ARC Lab, among other institutions, possessing 4D geometric control capabilities. Trained on the large-scale real-world dataset VerseControl4D, the model can handle complex dynamic scenes while maintaining strong spatiotemporal consistency. Users can specify camera and target trajectories to generate high-quality, geometrically consistent videos. The model has broad application prospects in video generation, virtual reality, and game development.

VerseCrafter's main functions

• 4D geometry controlUsers can achieve precise control over the viewpoint and object motion in videos by specifying camera trajectories and 3D Gaussian trajectories for multiple targets.
• Flexible control modesThe model supports separate camera control, separate target control, and joint control of camera and target to meet the needs of different scenarios.
• High-quality video generationWhile maintaining the realism of the video, ensure the geometric consistency of the generated video to avoid distortion.
• Multi-perspective consistencyThe model can generate consistent video content from different perspectives, making it suitable for multi-person interactive scenarios.
• Large-scale data supportTrained on the VerseControl4D dataset, covering both dynamic and static scenes, it improves the model's generalization ability.

VerseCrafter's technical principles

• Frozen Wan2.1 backbone networkThe model uses the pre-trained Wan2.1 as the base model, maintaining strong video generation and generalization capabilities, while injecting geometric control signals on the basis of Wan2.1.
• GeoAdapterA lightweight geometry adapter that encodes 4D control signals (camera trajectory and 3D Gaussian trajectory) into a multi-channel map and injects it into the diffusion block of Wan2.1 for precise control.
• 4D Control Signal RenderingThe camera trajectory and target trajectory are rendered as a background RGB/depth map and a 3D Gaussian trajectory map, which are then used as conditional signals input into the generative model.
• VerseControl4D DatasetBy extracting camera and target trajectories from large-scale real-world video data, it provides rich geometric supervision for model training and supports the generation of dynamic and static scenes.

VerseCrafter's project address

• Project official website: https://sixiaozheng.github.io/VerseCrafter_page/
• GitHub repositoryhttps://github.com/TencentARC/VerseCrafter
• HuggingFace model libraryhttps://huggingface.co/TencentARC/VerseCrafter
• arXiv technical paper: https://arxiv.org/pdf/2601.05138

Application scenarios of VerseCrafter

• Virtual Reality (VR) and Augmented Reality (AR)VerseCrafter can build immersive virtual worlds, allowing users to explore scenes in real time through camera and object motion control, enhancing the interactive experience.
• Game developmentGenerate dynamic backgrounds and object movements for games, optimize perspective switching and rendering effects, and reduce development costs.
• Video content creationCreators can use VerseCrafter to quickly generate high-quality dynamic videos to meet the creative needs of advertising, film, and animation production.
• Education and TrainingVerseCrafter can create realistic virtual teaching scenarios, such as historical reenactments and scientific experiment simulations, to enhance students' learning interest and engagement.
• Entertainment and MediaUsed to develop interactive video content, such as choice-based narrative videos, where viewers can change the course of the story by controlling the viewpoint and the movement of objects.