FaceLift - A single-image to 3D head model generation technology developed by Adobe in collaboration with the University of California.

What is FaceLift?

FaceLift, developed by Adobe and UC Merced, is a single-image to 3D head model conversion technology that can reconstruct a 360-degree head model from a single face image. FaceLift is based on a two-stage workflow: a diffusion-based multi-view generation model generates consistent side and back views from a single face image; the generated views are then fed into a GS-LRM reconstructor, producing a detailed 3D Gaussian representation. FaceLift accurately preserves individual identity features, generating 3D head models with fine geometric and textural details. FaceLift supports video input for 4D new view synthesis and can be seamlessly integrated with 2D facial re-animation technology to achieve 3D facial animation.

FaceLift's main functions

• Single Image 3D Head ReconstructionIt can quickly and efficiently reconstruct a 360-degree 3D head model from a single face image, including fine details of the face and hair.
• Multi-view consistencyThe generated 3D model remains consistent across different viewpoints, ensuring high-quality visuals from any angle.
• Identity preservationDuring the reconstruction process, the individual's identity characteristics are accurately preserved, maintaining a high degree of identity consistency even when generating invisible views.
• 4D New View CompositingSupports video input, enables 4D new view synthesis, generates consistent 3D models over time, and is suitable for dynamic scenes.
• Integration with 2D heavy animation technologySeamlessly integrates with 2D facial re-animation technology to achieve 3D facial animation, supporting digital entertainment and virtual reality applications.

FaceLift's technical principles

• Multi-view diffusion model:
  • Input processingBased on the image conditional diffusion model, this method uses a single frontal face image as input to generate images from multiple perspectives, including side and back views.
  • Diffusion model coreUsing the Stable Diffusion V2-1-unCLIP model, the embeddings generated by the CLIP image encoder are used as conditional signals to ensure identity consistency and multi-view consistency of the generated images.
  • Multi-view attention mechanismBased on a multi-view attention mechanism, the model shares information between different views to generate a consistent RGB image across multiple perspectives.
• GS-LRM Reconstructor:
  • Input fusionThe generated multi-view images and their corresponding camera poses are input into the GS-LRM model to generate a detailed 3D Gaussian representation.
  • 3D Gaussian representationUsing 3D Gaussian as the underlying representation, it is particularly suitable for capturing the complex details of the human head, such as hair. Each 2D pixel corresponds to a 3D Gaussian, with parameters including RGB color, scale, rotation quaternion, opacity, and ray distance.
  • Converter architectureGS-LRM employs a transformer architecture to regress pixel-aligned 3D Gaussians from a set of pose images, generating detailed 3D models.
• Optimization and Evaluation:
  • loss functionDuring training, the model is optimized using a combination of MSE and perceptual loss to ensure that the generated 3D model is visually highly consistent with the real image.
  • Evaluation indicatorsReconstruction quality was evaluated using standard metrics such as PSNR, SSIM, LPIPS, and DreamSim, and identity preservation was assessed based on ArcFace.

FaceLift's project address

• Project official website:https://www.wlyu.me/FaceLift/
• GitHub repository:https://github.com/weijielyu/FaceLift
• arXiv technical paper:https://arxiv.org/pdf/2412.17812

FaceLift application scenarios

• Virtual Reality (VR) and Augmented Reality (AR)Create realistic 3D virtual characters, provide an immersive experience, and support real-time interaction.
• Digital EntertainmentGenerate high-quality 3D character models for use in film, television, game development, and animation production, improving production efficiency and quality.
• Remote Existence SystemIn video conferencing and remote collaboration, presenting users in 3D enhances the naturalness and immersion of communication.
• Social media and content creationGenerate personalized 3D avatars, providing content creators with tools to enrich their creative content.
• Healthcare and EducationGenerate realistic 3D human body models for medical education, virtual surgical simulation, cultural heritage preservation, and virtual museum displays.