AB
AB
AiBoss

All Categories

project

Unique3D - An open-source image-to-3D generative model from Tsinghua University.

Unique3D is an open-source framework from Tsinghua University for converting single images into 3D models. By combining multi-view diffusion models and normal diffusion models, along with an efficient multi-level upsampling strategy, it can convert single images into 3D models...

What is Unique3D?

Unique3D is an open-source framework from Tsinghua University for converting single images to 3D models. By combining a multi-view diffusion model and a normal diffusion model, along with an efficient multi-level upsampling strategy, it can quickly generate high-fidelity and richly textured 3D meshes from a single image. Unique3D, combined with the ISOMER algorithm, further ensures the geometric and color consistency and accuracy of the generated 3D models. It can complete the conversion from a single-view image to a 3D model in just 30 seconds, producing results superior to image-to-3D model converters such as InstantMesh, CRM, and OpenLRM.

Unique3D Features

  • Single Image 3D Mesh GenerationUnique3D can automatically generate 3D mesh models from a single 2D image, transforming a planar image into a three-dimensional form with spatial depth.
  • Multi-view generationThe system uses a multi-view diffusion model to generate four orthogonal view images of the same object. These views capture the features of the object from different directions, providing comprehensive perspective information for 3D reconstruction.
  • Normal map generationUnique3D generates corresponding normal maps for each multi-view image. These maps record the orientation information of the object's surface, which is crucial for subsequent 3D model rendering. They can simulate how light interacts with the surface and enhance the realism of the model.
  • Multi-level resolution enhancementThe resolution of the generated image is gradually increased through a multi-level upsampling process, from low resolution to high resolution (e.g., from 256×256 to 2048×2048), making the texture and details of the 3D model clearer.
  • Integration of geometric and texture detailsDuring the reconstruction process, Unique3D tightly integrates color information and geometry to ensure that the generated 3D model is visually consistent with the original 2D image, while possessing complex geometric structures and rich texture details.
  • High-fidelity outputThe generated 3D model is highly consistent with the input 2D image in terms of shape, texture and color, and achieves high fidelity in both geometric accuracy and texture richness.

Unique3D's official website entrance

Unique3D's technical principles

  • Multi-view diffusion modelThis technique utilizes diffusion models to generate multi-view (typically four orthogonal views) images from single-view images. These models learn the distribution of 2D images through training and extend it to 3D space to generate images with different perspectives.
  • Normal diffusion modelIt works in conjunction with the multi-view diffusion model to generate corresponding normal maps for each generated view image. These normal maps contain the direction information of the surface normals, which is crucial for subsequent 3D reconstruction.
  • Multi-stage upsampling processA multi-level upsampling strategy is employed to gradually increase the resolution of the generated image. The initially generated image has a low resolution, which is then gradually increased to a higher resolution through upsampling techniques to obtain clearer details.
  • ISOMER mesh reconstruction algorithmAn efficient mesh reconstruction algorithm for reconstructing 3D meshes from high-resolution multi-view RGB images and normal maps. The ISOMER algorithm includes:
    • Initial mesh estimation: Quickly generate a rough topology and initial mesh for 3D objects.
    • Coarse-to-fine mesh optimization: Through an iterative optimization process, the shape of the mesh is gradually improved to make it closer to the target shape.
    • Explicit target optimization: Specify an optimization target for each vertex to address issues caused by inconsistent viewpoints and improve the accuracy of geometric details.
  • Color and geometric prior integrationDuring the mesh reconstruction process, color and geometric information are integrated into the mesh result to improve the visual realism and accuracy of the final model.
  • Explicit TargetDefine an optimization objective for each vertex. This is a mapping function from the vertex set to the color set, used to guide the optimization of vertex colors and improve the multi-view consistency of the model.
  • Extended Regularization: A technique used in the optimization process to avoid surface collapse and ensure the integrity of the model by moving vertices in the direction of their normals.
  • Color completion algorithmFor color completion in invisible areas, an efficient algorithm is used to smoothly propagate the color from the visible areas to the invisible areas, ensuring color consistency throughout the model.
Back