Hi3DGen - A 3D geometry generation framework jointly developed by the Chinese University of Hong Kong, ByteDance, and Tsinghua University.

What is Hi3DGen?

Hi3DGen is a high-fidelity 3D geometry generation framework jointly developed by researchers from the Chinese University of Hong Kong, Shenzhen, ByteDance, and Tsinghua University. It can generate high-fidelity 3D models from 2D images. By using normal maps as an intermediate representation, Hi3DGen can generate rich geometric details, significantly outperforming existing methods. The framework comprises three key components: an image-to-normal estimator, a normal-to-geometry learning method, and a 3D data synthesis pipeline.

Hi3DGen's main functions

• Generating high-fidelity 3D models from 2D imagesIt can convert 2D images into 3D geometric models with rich details.
• Image to Normal EstimationBy injecting noise and training two streams, low-frequency and high-frequency image modes are decoupled to achieve generalizable, stable and sharp normal estimation.
• Learning from normals to geometry: Enhance the fidelity of 3D geometry generation through latent diffusion learning based on normal regularization.
• 3D data synthesis: Build high-quality 3D datasets to support training.

Hi3DGen's technical principles

• Image to Normal EstimatorThe component decouples the low-frequency and high-frequency modes of an image through noise injection and dual-stream training. The low-frequency mode is responsible for the overall shape and structure, while the high-frequency mode is responsible for details and texture. It can generate generalizable, stable, and sharp normal maps, providing a high-quality intermediate representation for subsequent 3D geometry generation.
• Normal to Geometry Learning MethodThe latent diffusion model is trained using normal maps as a regularization method. This enhances the fidelity of 3D geometry generation, enabling the generated 3D models to retain more detail.
• 3D Data Synthesis PipelineThis feature utilizes a 3D data synthesis pipeline to construct high-quality 3D datasets for model training. It supports models in learning the mapping relationship from 2D images to 3D geometry.
• Two-stage generation processHi3DGen employs a two-stage generation process:
  • Phase 1: Basic Multi-View GenerationUsing a pre-trained video diffusion model, fine-tuned with additional camera pose conditions, single-view images are converted into low-resolution 3D perceptual sequence images (orbit videos).
  • Phase Two: 3D Perception Multi-View RefinementThe low-resolution multi-view images generated in the first stage are input into the 3D-perceived video video refiner to further improve the resolution and texture details of the images.
• 3D Gaussian Scattering (3DGS)Learn implicit 3D models from generated high-resolution multi-view images and render additional interpolated views via 3DGS.
• SDF-based reconstructionHigh-quality 3D meshes are extracted from enhanced dense views using an SDF (Signed Distance Function)-based reconstruction method.

Hi3DGen's project address

• Project official website:https://stable-x.github.io/Hi3DGen/
• Github repository:https://github.com/Stable-X/Hi3DGen

Application scenarios of Hi3DGen

• Game developmentQuickly generate high-quality 3D game assets, such as characters, props, and scenes.
• Film and television productionUsed to create realistic 3D effects and animations, saving time and costs associated with traditional modeling.
• 3D visualizationIt allows you to view and analyze 3D models from different angles, and is suitable for fields such as architectural design and industrial design.
• Virtual photographyGenerate high-quality images from different perspectives for online display and marketing.
• Cultural Relics Protection: Reconstructing 3D models from single photographs of cultural relics for digital preservation and research.
• Medical ImagingGenerate 3D models from medical images (such as X-rays and CT scans) to aid in diagnosis and treatment.