AB
AB
AiBoss

All Categories

project

LHM - An open-source tool from Alibaba Tongyi for generating animated 3D human models from a single image.

LHM (Large Animatable Human Reconstruction Model) is a product developed by Alibaba's Tongyi Labs that reconstructs an animated 3D human model from a single image. Based on a multimodal Transformer architecture, it integrates 3D geometry...

What is LHM?

LHM (Large Animatable Human Reconstruction Model) is a tool developed by Alibaba's Tongyi Labs that reconstructs animatable 3D human body model from a single image. Based on a multimodal Transformer architecture, it integrates 3D geometric features and 2D image features, uses an attention mechanism to preserve clothing geometry and texture details, and introduces a head feature pyramid encoding scheme to enhance facial detail recovery capabilities. LHM represents the reconstructed 3D model in the form of a 3D Gaussian point cloud, supporting real-time rendering and pose control animation. The model can generate a high-quality, animatable 3D human body model in seconds, suitable for immersive applications such as AR/VR.

LHM's main functions

  • Rapid ReconstructionIt can convert a single image into a 3D animated model in seconds without complicated post-processing.
  • High-fidelity detailsIt accurately preserves key information such as clothing texture and facial details to generate high-quality 3D models.
  • Real-time animationSupports real-time animation rendering based on posture control, suitable for immersive applications (such as AR/VR).
  • Strong generalization abilityIt performs exceptionally well in in-the-wild images and adapts to a variety of scenarios and poses.

LHM's technical principles

  • Multimodal Transformer architectureBased on the Transformer architecture, it fuses 3D geometric features (surface points sampled from SMPL-X templates) and 2D image features (extracted from a pre-trained visual Transformer) to effectively process geometric and visual information. A multi-scale feature extraction scheme designed for the head region aggregates features from different levels to enhance the ability to restore facial details.
  • 3D Gaussian point cloud representationThis technology uses 3D Gaussian point cloud representations to represent 3D models and supports real-time, high-quality rendering. The network directly predicts the parameters of the Gaussian point cloud (such as position, rotation, scaling, and color), enabling rapid conversion from input images to 3D models.
  • Self-supervised learningThe model is trained on large-scale video data and optimized using rendering loss and regularization terms, without relying on scarce 3D scan data. During training, regularization terms of "as close as possible" and "as spherical as possible" are introduced to maintain the geometric plausibility of the 3D model.
  • Real-time animation supportBased on SMPL-X skeleton parameters, the reconstructed 3D model is deformed to the target pose, supporting real-time pose control animation. The entire reconstruction and animation process is completed in a single forward propagation, making it suitable for real-time applications.

LHM's project address

Application scenarios of LHM

  • Virtual Reality (VR) and Augmented Reality (AR)Quickly transform photos into animated 3D virtual characters, enhancing immersion and interactivity.
  • Game developmentIt can quickly generate high-quality 3D character models, support real-time animation, and improve development efficiency and game experience.
  • Film and television productionUsed in special effects production and animated films, it quickly generates character models, improving production efficiency and quality.
  • Social media and content creationUsers can generate 3D virtual avatars for social media, and creators can quickly generate 3D characters for short videos, etc.
  • Education and trainingCreate virtual teachers or teaching assistants for online education, and generate 3D models for simulation training in fields such as medicine and the military.
Back