DimensionX - A framework for generating complex 3D and 4D scenes from a single image, jointly developed by HKUST, Tsinghua University, and Biodata Technology.

What is DimensionX?

DimensionX is a framework jointly developed by the Hong Kong University of Science and Technology, Tsinghua University, and BioNTech. It can generate highly realistic 3D and 4D scenes from single images, achieving precise control over spatial and temporal dimensions based on video diffusion technology. The framework decouples spatial and temporal factors based on ST-Director technology, supporting independent or combined control to generate complex scenes with dynamic changes. DimensionX includes trajectory awareness mechanisms and identity-preserving denoising strategies to enhance scene consistency and realism.

DimensionX's main functions

• 3D scene generationGenerate new perspective renderings from a single image to construct a 3D scene.
• 4D scene generationGenerate dynamic scenes that include temporal and spatial changes from a single image.
• Video diffusion controlBased on ST-Director technology, it achieves decoupling and precise control of spatial and temporal factors during video diffusion.
• Trajectory perception mechanism: Designed for 3D generation, handling complex real-world scenes and camera movement.
• Identity Preservation Denoising Strategy: Designed for 4D generation, enhancing scene consistency, especially between dynamic objects and backgrounds.

DimensionX's technical principles

• ST-Director (Space and Time Director):
  • Dimensional Awareness LoRAsLearn low-rank adaptations (LoRAs) from dimensionally varied data to decouple spatial and temporal factors in video diffusion.
  • S-Director (Spatial Director): Responsible for generating video frames related to spatial changes and controlling the camera's viewpoint and position.
  • T-Director (Time Director): Responsible for generating video frames that change over time and controlling the dynamics of objects in the scene.
• Dimensional Perception DecompositionDefine spatial and temporal equivalence relationships, create S-Quotient Space and T-Quotient Space, and capture spatial trajectories and temporal motion trajectories in the video, respectively.
• Dimension-aware assemblies that do not require trainingBased on the denoising mechanism in the video diffusion process, a training-free method is developed to achieve hybrid dimension control, which is used to switch between S-Director and T-Director at different stages of the denoising process to generate videos containing spatial and temporal variations.
• 3D scene generation:
  • Trajectory perception mechanism: Train multiple S-Directors based on different camera motion trajectories to cover a wide range of camera motion modes.
  • Video interpolation modelGenerates high-quality interpolated videos with smooth and consistent transitions in sparse views.
• 4D scene generation:
  • Reference video latent sharing: Enhance consistency among all spatial variant videos by selecting a reference frame and sharing its latent code.
  • Appearance refinement: Refine the dynamic video from each viewpoint to enhance the stability and consistency between multi-view videos.

DimensionX project address

• Project official website:chenshuo20.github.io/DimensionX
• GitHub repository:https://github.com/wenqsun/DimensionX
• arXiv technical paper:https://arxiv.org/pdf/2411.04928

Application scenarios of DimensionX

• Film and entertainment industryGenerate special effects scenes, reduce shooting costs, and create virtual backgrounds.
• Game developmentGenerate realistic game environments and quickly prototype.
• Virtual Reality (VR) and Augmented Reality (AR)Create dynamic 3D environments to enhance the user experience.
• Architecture and Urban PlanningGenerate architectural models based on design drawings to simulate urban planning.
• Education and trainingCreate visualizations of historical scenarios to simulate emergency situations.