FantasyWorld - A 3D world modeling framework jointly launched by Amap and Beijing University of Posts and Telecommunications

What is FantasyWorld?

FantasyWorld is an innovative 3D world modeling framework developed in collaboration between AMAP (Amap) and Beijing University of Posts and Telecommunications. It focuses on generating high-quality 3D scenes through unified video and 3D prediction. The framework achieves joint modeling of latent video variables and implicit 3D fields by adding trainable geometric branches to a frozen video base model, enabling the generation of geometrically consistent 3D-aware videos in a single forward propagation. FantasyWorld excels in multi-view consistency, maintaining high visual realism and geometric consistency even under large viewpoint changes (such as 180-degree rotation).

Main functions of FantasyWorld

• Joint modeling video and 3D sceneBy adding a trainable geometric branch to the frozen video base model, FantasyWorld can simultaneously generate videos and 3D scenes, enabling joint modeling of video latent variables and implicit 3D fields, and providing a general representation for downstream 3D tasks.
• Cross-branch supervision mechanismThe method introduces geometric cues to guide video generation, while using video priors to constrain 3D prediction, thereby achieving interactive optimization of video and 3D information and generating consistent and generalizable 3D-perceptual videos.
• Multi-view consistency optimizationEven with large changes in perspective (such as a 180-degree rotation), the generated video can still maintain a high degree of multi-view coherence and geometric fidelity, ensuring the stability of 3D scenes under different perspectives.
• High-efficiency forward propagation architectureEmploying preprocessing modules (PCBs) and integrated reconstruction and generation (IRG) modules, video and geometric features are optimized through multimodal conditions to ensure high-quality 3D scene generation in a single forward propagation.
• Supports multiple downstream applicationsIt provides a strong technological foundation for applications that require 3D world modeling, such as AR/VR content creation and robot navigation, and promotes the further development of spatial intelligence and artificial intelligence.

The technical principles of FantasyWorld

• Geometric Enhancement Video Base Model: Add a trainable geometric branch to the frozen video base model to achieve joint modeling of video latent variables and implicit 3D fields, enabling the model to handle video generation and 3D geometric inference simultaneously.
• Cross-branch supervision mechanismThe method guides video generation through geometric cues and utilizes video priors to constrain 3D prediction, thereby optimizing the interaction between video and 3D information to generate consistent and generalizable 3D-perceptual videos.
• Multimodal data fusionBy combining video data and geometric information, the output of the model is optimized using multimodal conditions to ensure that the generated 3D scene has a high degree of coherence and consistency in both vision and geometry.
• High-efficiency forward propagation architectureEmploying preprocessing modules (PCBs) and integrated reconstruction and generation (IRG) modules, high-quality 3D scene generation is ensured in a single forward propagation by iteratively optimizing video latent variables and geometric features.
• Lightweight adapters and cross-attentionA lightweight adapter and cross-attention mechanism are introduced between the geometry branch and the video branch to enable information exchange and collaborative optimization between the two branches.

FantasyWorld project address

• Project official website: https://fantasy-amap.github.io/fantasy-world/
• Github repository: https://github.com/Fantasy-AMAP/fantasy-world
• arXiv technical paper: https://arxiv.org/pdf/2509.21657

Application scenarios of FantasyWorld

• AR/VR content creationIt generates high-quality 3D world models, providing realistic virtual environments for augmented reality (AR) and virtual reality (VR) applications, and supporting the development of immersive experiences.
• Robot NavigationIt provides robots with accurate 3D environmental perception and path planning support, helping them better understand spatial structures and achieve more intelligent navigation and interaction.
• Flying Street ViewMerchants can upload mobile phone videos to generate high-fidelity 3D virtual street views, allowing users to experience the layout of the venue, such as restaurant seating areas, in advance.
• Digital twin: To build digital twin models of cities for use in urban planning, architectural design and infrastructure management, providing high-precision 3D visualization and simulation.
• Game developmentIt provides game developers with the ability to quickly generate high-quality 3D game scenes, reducing development costs and enhancing the visual effects of games.