IC-Portrait - A personalized portrait generation framework launched by ETH Zurich in collaboration with Zhejiang University and other institutions.
IC-Portrait is a novel personalized portrait generation framework that addresses the challenges posed by the diversity of user profile images (such as differences in appearance and lighting conditions). It breaks down the portrait generation task into "light-aware stitching" and "viewpoint consistency..."
What is IC-Portrait?
IC-Portrait is a novel personalized portrait generation framework that addresses the challenges posed by the diversity of user profile images (such as differences in appearance and lighting conditions). It achieves high-fidelity identity preservation and viewpoint consistency by decomposing the portrait generation task into two sub-tasks: "lighting-aware stitching" and "viewpoint consistency adaptation." IC-Portrait utilizes a high-ratio masking autoencoder technique (approximately 80% of the input image is masked) for self-supervised lighting feature learning and learns contextual correspondences using a synthetic viewpoint consistency dataset.
Main functions of IC-Portrait
- Identity PreservationIC-Portrait focuses on accurately preserving individual identity features during the generation process. By decomposing the portrait generation task into two sub-tasks—light-aware stitching and viewpoint consistency adaptation—the framework significantly improves the fidelity and stability of identity preservation.
- 3D-Aware RelightingIC-Portrait demonstrates 3D-aware relighting capabilities, enabling the generation of high-quality portraits under varying lighting conditions. This ensures the generated portraits maintain a consistent viewpoint and adapt to diverse lighting conditions.
- Compatibility with existing production pipelinesIC-Portrait generates reference features that are compatible with ControlNet, allowing for easy integration into existing generation pipelines. This enables the framework to seamlessly interface with existing image generation tools, facilitating user integration into existing workflows.
The technical principle of IC-Portrait
- View-Consistent AdaptationIC-Portrait learns contextual correspondences through a synthetic viewpoint-consistent dataset, enabling it to warp reference portraits to arbitrary poses and providing robust spatial alignment viewpoint conditions. This ensures that the generated portraits maintain consistency across different viewpoints.
- Lighting-Aware StitchingThe framework is based on a high-ratio masking autoencoder technique (approximately 80% of the input image is masked) and learns the lighting features of the reference image through self-supervised learning. This effectively reduces the adaptation gap between the user's data image and the reference image, while preserving local lighting cues and global shadow effects.
IC-Portrait project address
- arXiv technical paper:https://arxiv.org/pdf/2501.17159
Applications of IC-Portrait
- Social media and personal brandingUsers can use IC-Portrait to generate personalized portraits with different perspectives and lighting conditions for use as social media profile pictures or personal brand promotional materials.
- Artistic Creation and DesignArtists and designers can use IC-Portrait to generate portrait works with specific styles or lighting effects, exploring different forms of artistic expression.
- Virtual try-on and fashion retailIn the fashion industry, IC-Portrait can combine virtual try-on technology, allowing users to see how they look in different outfits, providing a more personalized and convenient shopping experience.
- Gaming and Virtual Reality (VR)IC-Portrait can generate virtual characters with specific identity characteristics, enhancing interactivity and immersion in games and VR environments.