AB
AB
AiBoss

All Categories

project

GeneMAN - A 3D human body model creation framework jointly developed by Shanghai AI Lab and Peking University and other universities.

GeneMAN is a 3D human creation framework jointly developed by the Shanghai AI Lab, Peking University, Nanyang Technological University, and Shanghai Jiao Tong University. It can create high-fidelity 3D human models from a single image. The framework does not rely on parametric human models...

What is GeneMAN?

GeneMAN is a 3D human creation framework jointly developed by the Shanghai AI Lab, Peking University, Nanyang Technological University, and Shanghai Jiao Tong University. It can create high-fidelity 3D human models from a single image. The framework does not rely on parametric human models; instead, it trains human-specific 2D and 3D prior models using multi-source, high-quality human datasets, including 3D scans, multi-view videos, single photographs, and synthetic data. GeneMAN achieves its goal of extracting high-quality 3D human models from natural environment image data based on geometric initialization, sculpting processes, and multi-spatial texture refinement techniques, regardless of variations in human proportions, poses, or clothing within the image.

GeneMAN's main functions

  • Handling diverse human proportionsRegardless of whether the person in the image is in full-body, half-body, or close-up, GeneMAN can reconstruct the corresponding 3D model.
  • Adapt to different clothing and posturesGeneMAN can handle a variety of clothing and body poses, including natural poses and common objects.
  • Processing field imagesGeneMAN excels at processing images from natural environments, also known as "field data," which may contain a variety of complex backgrounds and lighting conditions.
  • Generate high-quality texturesBased on a multi-space texture refinement process, GeneMAN can generate 3D human body textures that are rich in detail and consistent with the input image.

GeneMAN's technical principles

  • Training 2D and 3D Human Prior Models:
    • Text-to-image diffusion modelTrain a human-specific text-to-image diffusion model to generate 2D human priors.
    • View Condition Diffusion ModelTrain a view-conditional diffusion model to generate 3D human priors.
  • Geometric initialization and carving process:
    • Template-free geometry initializationInitialize a template-free 3D geometry using NeRF (Neural Radiation Field) technology.
    • Combining prior knowledge and loss: Combine GeneMAN's 2D and 3D priors with structured differential loss (SDS loss) to guide geometry initialization.
    • Reference lossUse reference loss to ensure alignment with the input image.
    • High resolution refinement: Convert NeRF to DMTet (Depth Polyhedral Mesh) for high-resolution geometric refinement, guided by a pre-trained human-specific normal and depth-adaptive diffusion model.
  • Multi-space texture refinement process:
    • Coarse texture generationGenerate a rough texture using multi-view texture technology.
    • Potential space refinementIteratively refine the texture in the latent space.
    • Pixel space refinement: Detailed textures are obtained in pixel space based on optimized UV maps, and optimized using ControlNet based on 2D priors.

GeneMAN's project address

Applications of GeneMAN

  • Virtual try-onUsers upload their full-body photos to generate 3D models for online clothing try-on, providing a more realistic fitting experience.
  • Games and entertainmentIn game and film production, create or modify character models to provide more personalized and realistic character designs.
  • Augmented Reality (AR) and Virtual Reality (VR)In AR and VR applications, generate 3D avatars of users to enhance immersion and interactivity.
  • Fashion DesignDesigners can quickly convert design sketches into 3D models, accelerating the design process.
  • Fitness and health managementBased on the analysis of the user's 3D model, it provides personalized fitness guidance and health advice.
Back