Pixel3DMM - A 3D face reconstruction framework developed by Munich in collaboration with University College London and others.

What is Pixel3DMM?

Pixel3DMM is a single-image 3D face reconstruction framework jointly developed by the Technical University of Munich, University College London, and Synthesia. Based on the DINO model, the framework introduces a specialized prediction head to accurately reconstruct the geometry of a 3D face from a single RGB image. Pixel3DMM performs exceptionally well on multiple benchmarks, significantly outperforming existing methods in handling complex facial expressions and poses. Pixel3DMM introduces new benchmarks covering diverse facial expressions, viewpoints, and ethnicities, providing new evaluation standards for research in this field.

Pixel3DMM's main features

• High-precision 3D face reconstructionAccurately reconstruct the geometry of a 3D human face from a single RGB image, including facial shape, expression, and pose.
• Handling complex expressions and gesturesIt excels at processing complex facial expressions and non-frontal view facial images, and can effectively reconstruct high-quality 3D facial models.
• Decoupling identity and expressionThe goal is to recover neutral facial geometry from posed images, which means that facial identity and expression information can be distinguished and reconstructed.

The technical principles of Pixel3DMM

• Pre-trained Vision TransformerDINOv2 was used as the backbone network to extract features from the input image. DINOv2 is a powerful self-supervised learning model that can extract rich semantic features, providing a solid foundation for subsequent geometric cue prediction.
• Prediction HeadBuilding upon the DINOv2 backbone network, additional transformer blocks and upper convolutional layers are added to increase the feature map resolution to the desired size, ultimately outputting predicted geometric cues. These geometric cues, including surface normals and UV coordinates, provide crucial constraint information for optimizing the 3D face model.
• FLAME model fittingThis study optimizes the parameters of the FLAME model to reconstruct 3D faces based on predicted surface normals and UV coordinates. FLAME is a parametric 3D face model capable of representing facial identity, expression, and pose. By minimizing the difference between the predicted geometric cues and the rendered results from the FLAME model, the parameters of the FLAME model are optimized to achieve high-precision 3D face reconstruction.
• Optimization strategyDuring the inference phase, the parameters of the FLAME model are optimized based on minimizing the difference between the predicted geometric cues and the rendering results of the FLAME model.
• Data preparation and trainingWe trained a prediction network using multiple high-quality 3D face datasets (such as NPHM, FaceScape, and Ava256) and unified them into the topology of the FLAME model based on non-rigid registration. The datasets cover a variety of identities, expressions, viewpoints, and lighting conditions to ensure the model's generalization ability.

Pixel3DMM project address

• Project official website:https://simongiebenhain.github.io/pixel3dmm/
• arXiv technical paper:https://arxiv.org/pdf/2505.00615

Application scenarios of Pixel3DMM

• Film and GamesIt can quickly generate high-quality 3D face models, improve expression capture and animation effects, and reduce costs.
• VR/ARCreate realistic virtual avatars to enhance immersion and realism of interaction.
• Social VideoGenerate virtual backgrounds and special effects to enhance visual appeal and enable more accurate facial expression recognition and interaction.
• Medical AestheticsIt assists in facial surgery planning and provides virtual makeup and beauty effect previews.
• academic researchIt provides new methods and benchmarks to drive the development of 3D face reconstruction technology.