RealisHuman - A post-processing framework for repairing deformed human body parts in generated images.
RealisHuman is an innovative post-processing framework that refines human body parts, such as hands and faces, in generated images. The framework achieves this through a two-stage approach. First, using the original deformed area as a reference, it generates a more refined version...
What is RealisHuman?
RealisHuman is an innovative post-processing framework that refines human body parts, such as hands and faces, in generated images. The framework achieves this through a two-stage approach. First, using the original deformed area as a reference, it generates realistic human body parts to ensure consistency with the original image. Second, by redrawing the surrounding area, it seamlessly integrates the corrected human body parts back into their original positions, ensuring a smooth and realistic blend. RealisHuman improves the realism of human body parts in generated images. The framework also has strong generalization capabilities, handling images of different styles, including cartoons and sketches.
RealisHuman's main functions
- Generate realistic human body partsRealisHuman uses the original deformed parts as a reference to generate realistic human body parts, such as hands and faces, ensuring that the generated parts are consistent with the original image in detail.
- Seamless integration of correction partsAfter generating realistic body parts, RealisHuman seamlessly integrates the corrected human body parts back into the original image by redrawing the surrounding area, ensuring a smooth and realistic blend.
- Processing multiple image stylesIt has strong generalization ability and can effectively handle images of different styles, including cartoons and sketches.
- Improve the realism of generated imagesThrough the two-stage method described above, RealisHuman improves the realism of human body parts in generated images.
RealisHuman's technical principles
- Phase 1: Generating realistic human body parts
- Reference to the original malformation siteRealisHuman uses the original deformed areas as a reference to ensure that the generated human body parts are consistent with the original image in detail.
- Extract detailed informationBy using tools such as Part Detail Encoder and DINOv2, detailed information about the original deformed areas, including skin color and texture, can be extracted.
- 3D pose estimationThe 3D pose estimation results extracted from the original deformed parts are used to guide the generation of human body part images to ensure the accuracy and realism of the pose.
- Phase Two: Seamless Integration of Corrected Parts
- Positioning and croppingFirst, return the corrected human body parts to their original positions and cover the surrounding areas.
- Repair and fusionTreat the process as a repair problem and train a model to seamlessly blend human body parts with the surrounding area, ensuring a smooth transition and realistic fusion.
- RedrawDuring the integration process, the area between the background and the corrected human body parts was redrawn to avoid collage marks.
RealisHuman's project address
- Github repository:https://github.com/Wangbenzhi/RealisHuman
- arXiv technical paper:https://arxiv.org/pdf/2409.03644
RealisHuman Application Scenarios
- Game developmentIn game character design, RealisHuman can be used to generate realistic character hand and facial details, enhancing the realism and immersion of the character.
- Film and television productionIn film and television special effects production, it can help repair or enhance a character's facial expressions and hand movements, making the special effects visuals more natural and realistic.
- Image restorationFor images with hand or facial deformities, RealisHuman can be used to repair these areas, making them look more natural.
- Artistic CreationIn artistic creation, artists can use this technology to generate human body parts with unique styles and details, enriching the expressiveness of their artworks.
- Interactive applicationsIn applications such as virtual try-on or virtual makeup, it can ensure that the details of the virtual character's hands and face are consistent with the user's real features.