PSHuman - an open-source single-image 3D portrait reconstruction technology that requires only one photo.

What is PSHuman?

PSHuman is an advanced single-image 3D human portrait reconstruction technology. Based on a cross-scale multi-view diffusion model, it can generate highly realistic 3D human portrait models, including detailed facial expressions and full-body poses, from a single image. Its core advantage lies in its ability to simultaneously model the joint probability distribution of global shape and local details, avoiding geometric distortion and maintaining consistency in body shape across different views. PSHuman efficiently recovers realistic textured human meshes through explicit human sculpting technology. It performs exceptionally well on multiple datasets, demonstrating outstanding geometric detail, texture fidelity, and generalization ability.

PSHuman's main functions

• Single-image 3D portrait reconstructionUsers only need to provide a portrait photo, and PSHuman can quickly generate a detailed 3D model, greatly improving the efficiency of 3D human body modeling.
• Multi-view diffusion technologyBased on an advanced multi-view diffusion model, it generates human images from multiple angles from a single-view photograph, ensuring that each angle reflects the original object as realistically as possible, thus solving the problem of single-view image reconstruction.
• High-fidelity facial detailsBy employing a cross-scale diffusion method, which takes into account both the overall body shape and the details of local facial features, the problem of geometric deformation is avoided. The resulting 3D model exhibits excellent facial features with a high degree of realism and detail.
• Combined with SMPL-X human body modelIt combines SMPL-X parametric human body modeling, which can represent human bodies in different postures and body types, generating more natural and realistic human body movements and forms, further improving the realism of 3D models.
• Background removalSupports the use of Clipdrop or rembg tools to remove the background of portrait photos, simplifying the subsequent processing workflow.
• Structured outputThe generated 3D models and rendered videos are saved in the form of structured files, making it easy for users to view and share them.
• SMPL-free versionIt can generate multiple views without SMPL conditions, and is suitable for portraits in general poses, thus expanding its application scenarios.

PSHuman's technical principles

• Body + facial enhancement and multi-view generationThe system first feeds the input photo and a predicted human skeleton model (SMPL-X) into a multi-view image diffusion model. The model can generate full-body images from six different angles, as well as high-precision local facial images, ensuring that the details of the whole body and face are as realistic as possible.
• Cross-scale diffusion methodsIt can simultaneously model the joint probability distribution of global body shape and local facial features, avoiding the problem of geometric deformation, so that the final generated 3D model has excellent facial features, with high realism and detail.
• Explicit figure sculpting based on SMPL-XBased on the generated normal map (showing 3D shape details) and color map (texture details), the SMPL-X model is adjusted, deformed, and refined using differentiable rendering techniques. This step combines multi-view generation and 3D sculpting techniques, enabling PSHuman to efficiently "sculpt" realistic 3D character models from a single photograph.
• Multi-view image fusionDifferentiable rendering technology is used to fuse multi-view color images, reducing generation inconsistencies. The fused texture is then projected onto the reconstructed 3D mesh to achieve texture mapping.
• Optimize and rebuild modulesGeometric optimization is performed by aligning multi-view images through optimized SMPL-X parameters. Differentiable rasterization techniques are used to sculpt the mesh to match the observed normal image.

PSHuman's project address

• Project official website:https://penghtyx.github.io/PSHuman
• GitHub repository:https://github.com/pengHTYX/PSHuman
• arXiv technical paper:https://arxiv.org/pdf/2409.10141

Application Scenarios of PSHuman

• Film and television production and game developmentIn film and television special effects and game scenes, PSHuman can quickly generate realistic 3D character models, improve production efficiency, reduce production costs, and add more realism and detail to film and television and games.
• Virtual Reality and Augmented RealityIn the VR and AR fields, 3D human models generated by PSHuman allow users to interact with virtual characters in virtual environments, providing an immersive experience. Furthermore, it can be applied to virtual try-on, virtual makeup, and other services, offering users convenient and personalized options.
• Fashion DesignThe fashion industry can use PSHuman technology to quickly generate clothing try-on effects for clothing design and display, accelerating the design process and improving user experience.
• Personalized customizationIt generates personalized 3D models from a single photo, which can be used to customize personalized products, such as custom dolls or game characters, to meet users' needs for personalized and customized content.