PortraitGen - An AI-powered portrait video editing tool developed by the University of Science and Technology of China
PortraitGen is an AI-powered portrait video editing tool developed by a research team at the University of Science and Technology of China. Based on 3D Gaussian sputtering technology and a neural Gaussian texture mechanism, it converts 2D portrait videos into 4D Gaussian fields, achieving high-quality 3D...
What is PortraitGen?
PortraitGen is an AI-powered portrait video editing tool developed by a research team at the University of Science and Technology of China. Based on 3D Gaussian sputtering technology and a neural Gaussian texture mechanism, it converts 2D portrait videos into 4D Gaussian fields, achieving high-quality 3D and temporally consistent editing. The tool supports multimodal editing, including text-driven, image-driven editing, and relighting, enabling fast and efficient stylization, clothing changes, and lighting adjustments for people in videos. Through face-aware editing and expression similarity guidance, PortraitGen ensures that the edited portraits are natural and consistent with the original video frames.
PortraitGen's main functions
- Multimodal portrait editingIt supports both text-driven and image-driven editing modes. Users can specify character actions, expressions, and scene changes by entering text descriptions, or use reference images to achieve style transfer or virtual try-on.
- RelightingBased on IC-Light technology, it dynamically adjusts the lighting effects in the video based on text descriptions, so that the lighting blends naturally with the scene.
- Face perception editingThe facial recognition editing module ensures that the facial structure and personalized features of the person are preserved during the video editing process, and optimizes the naturalness and consistency of the person's expressions and facial movements.
- Style switching and virtual try-onIt supports style transfer and virtual try-on, allowing users to transfer global styles (such as converting to animation style) using reference images, or add virtual clothing and other items to characters in videos.
- Multi-camera and complex scene processingIt can handle multi-camera videos while maintaining style and character consistency. It can also achieve complex video rendering styles, such as Lego style or pixel art style, through Gaussian texture technology.
- Fast generation and high frame rate outputIt can complete editing in a short time and supports a rendering speed of up to 100 frames per second (FPS) when generating videos, making it suitable for high-efficiency video production scenarios.
PortraitGen's technical principles
- 3D Gaussian Sputtering (3DGS)Based on a 3D Gaussian representation of the scene, the center point, direction, and size of each Gaussian, as well as its opacity and color attributes, are defined by a 3D covariance matrix to construct a dynamic 3D field.
- Neural Gaussian Texture MechanismMaintain a 3D Gaussian field in the UV space of the SMPL-X model, deform the Gaussian field according to the deformation of the underlying mesh of the input video tracking, and convert the feature map into RGB signal using UV mapping and a 2D neural renderer.
- Facial perception editing moduleThe head area is edited twice to enhance facial structure perception and improve editing quality.
- Expression similarity guidanceThe rendered image and the input source image are mapped to the latent expression space of EMOCA, and the similarity of expressions is ensured by a loss function.
- Multimodal editing technologyBy combining knowledge from large-scale 2D generative models, text-driven editing, image-driven editing, and relighting can be achieved.
PortraitGen's project address
- Project official websiteustc3dv.github.io/PortraitGen
- GitHub repository:https://github.com/USTC3DV/PortraitGen-code
- arXiv technical paper:https://arxiv.org/pdf/2409.13591
Applications of PortraitGen
- Film and television productionIn film, television series, or short film production, create or modify the appearance of characters to achieve special effects makeup or stylized scene transitions.
- Artistic CreationArtists and illustrators use PortraitGen to create portrait artworks with specific styles, such as converting portraits into pixel art or oil painting styles.
- Advertising and promotionIn the advertising industry, portraits of individuals are customized based on brand image or product characteristics to attract the target audience.
- fashion industryFashion designers and retailers are using virtual try-on features to showcase clothing and accessories in a virtual environment, providing customers with a new shopping experience.
- Social media and short videosContent creators and influencers use PortraitGen to edit their own portrait videos, adding creative effects and increasing the appeal and interactivity of their content.
- Game developmentIn game character design, it allows for the rapid generation or editing of character appearances, enhancing the personalization and richness of the game.