HoloTime - A panoramic 4D scene generation framework jointly developed by Peking University and Pengcheng Laboratory
HoloTime is a panoramic 4D scene generation framework developed by Peking University Shenzhen Graduate School and Pengcheng Laboratory. Based on a video diffusion model, it transforms a single panoramic image into a panoramic video with realistic dynamic effects, which is then further reconstructed into...
What is HoloTime?
HoloTime is a panoramic 4D scene generation framework developed by Peking University Shenzhen Graduate School and Pengcheng Laboratory. Based on a video diffusion model, it transforms a single panoramic image into a panoramic video with realistic dynamic effects, further reconstructing it into an immersive 4D scene. HoloTime incorporates the 360World dataset, containing a large number of panoramic videos captured by fixed cameras, used to train the Panoramic Animator to generate high-quality panoramic videos. HoloTime introduces Panoramic Space-Time Reconstruction technology, which converts panoramic videos into 4D point clouds based on spatiotemporal depth estimation, optimizing them into a consistent 4D Gaussian point cloud representation to achieve an immersive virtual reality experience.
HoloTime's main functions
- Generate panoramic video from a single panoramic imageIt transforms static panoramic images into panoramic videos with dynamic effects, containing rich motion information such as object movement and scene changes.
- Reconstruction of panoramic video into 4D scenesIt supports converting generated panoramic videos into 4D point clouds, further optimizing them into a consistent 4D scene representation, and supporting virtual roaming and multi-view observation.
- Immersive experience supportThe generated 4D scenes can provide an immersive interactive experience for VR (virtual reality) and AR (augmented reality) applications, allowing users to move and explore freely within the scene.
HoloTime's technical principles
- Panoramic Animator:
- Two-stage generation strategyFirst, a low-resolution coarse video is generated to provide global motion guidance; then, a high-resolution refinement model is used to enhance local details.
- Hybrid Data Fine-tuning (HDF)The model is trained by combining panoramic video and ordinary video data of similar landscapes to compensate for differences in data distribution and improve the generalization ability of the model.
- Panoramic Circular Techniques (PCT)Create repeating regions on the left and right sides of the video for blending to ensure the horizontal continuity of the panoramic video and avoid visual breaks at the stitching points.
- Panoramic Space-Time ReconstructionThis approach uses a panoramic optical flow estimation model and a narrow field-of-view depth estimation model to estimate the depth of each frame of panoramic video, ensuring the temporal and spatial continuity of depth information. The panoramic video and its depth map are then converted into a 4D point cloud with temporal attributes, serving as the initial representation of the 4D scene. Based on the optimized 4D point cloud representation, spatially and temporally consistent 4D scene reconstruction is achieved, supporting efficient rendering and dynamic view composition.
- 360World DatasetThis provides a large-scale fixed-camera panoramic video dataset for training the Panoramic Animator. The dataset contains rich scene and dynamic information, supporting the model in learning the generation rules of panoramic videos.
HoloTime's project address
- Project official website:https://zhouhyocean.github.io/holotime/
- GitHub repository:https://github.com/PKU-YuanGroup/HoloTime
- HuggingFace model library:https://huggingface.co/Marblueocean/HoloTime
- arXiv technical paper:https://arxiv.org/pdf/2504.21650
Application scenarios of HoloTime
- Virtual Reality (VR) and Augmented Reality (AR)It provides immersive 4D scenes, allowing users to roam freely in a virtual environment and enhancing their experience.
- Virtual tourism and online exhibitionsGenerate panoramic 4D scenes, allowing users to remotely tour attractions or exhibitions as if they were actually there.
- Film and television productionIt can quickly generate high-quality panoramic backgrounds and special effects, reduce shooting costs, and enhance visual effects.
- Game developmentCreate dynamic game scenes to enhance player immersion and visual experience.
- Architectural design and urban planningGenerate panoramic 4D scenes to help designers visually showcase their design plans and assess the effects in advance.