PersonaCraft - A technology developed by Seoul National University for generating multi-identity full-body images from a single reference image.
PersonaCraft is a personalized full-body image synthesis technology developed by Seoul National University in South Korea. Combining diffusion models and 3D human modeling, it can generate realistic, personalized full-body images of multiple people from a single reference image. PersonaCraft...
What is PersonaCraft?
PersonaCraft, developed by Seoul National University in South Korea, is a personalized full-body image synthesis technology. Combining diffusion models and 3D human modeling, it can generate realistic, personalized full-body images of multiple people from a single reference image. PersonaCraft effectively handles occlusion issues between people and supports user-defined body shape adjustments for more flexible personalization. Based on precise control of body posture and shape, PersonaCraft excels in generating high-quality images in complex scenes, setting a new standard for multi-person image synthesis.
The main functions of PersonaCraft
- Multi-person image synthesisGenerate realistic images containing multiple people based on a single reference image.
- ObscuringEffectively manage the occlusion problem between people to ensure that the body parts of each person in the image are accurately displayed.
- Full-body personalizationIt not only focuses on personalizing facial identity, but also accurately personalizes each person's overall body shape.
- User-defined body shape controlUsers can adjust the body proportions and shape of the generated character according to their personal preferences.
- 3D perception posture condition controlUsing SMPLx-ControlNet (SCNet) for 3D perception pose condition control improves the accuracy of body shape and pose.
The technical principles of PersonaCraft
- Combining 3D human modeling with diffusion modelPersonaCraft integrates 3D human modeling (especially SMPLx models) and diffusion models to enhance control over the shape and posture of the character's body.
- SMPLx-ControlNet (SCNet)Using depth maps generated by the SMPLx model as conditional signals, we can accurately capture body shape and pose, and effectively handle complex occlusion problems.
- Facial and body identity extractionBased on technologies such as InsightFace, facial identity embeddings are extracted from reference images, and body shape parameters are extracted using the SMPLx fitting method.
- 3D Perceived Pose ConditionsUnlike traditional 2D skeleton pose conditions, PersonaCraft uses 3D pose information to provide a more comprehensive representation of human pose, including depth information.
- Multi-person personalized image synthesisCombining SCNet and IdentityNet, facial masks are used to accurately locate identities, enabling personalized image synthesis for multiple identities.
PersonaCraft's project address
- Project official website:gwang-kim.github.io/persona_craft
- GitHub repository:https://github.com/gwang-kim/PersonaCraft
- arXiv technical paper:https://arxiv.org/pdf/2411.18068
Application scenarios of PersonaCraft
- social mediaUsers can customize their own or their friends' virtual avatars according to their preferences, and use them as social media profile pictures, representative images in virtual spaces, etc.
- Advertising and MarketingBrands create personalized advertising images that match the preferences of their target audience, thereby increasing the appeal of their ads and resonating with users.
- Fashion and ClothingIn the fashion industry, showcasing how clothing looks on people of different body types and backgrounds provides a more personalized shopping experience.
- Games and EntertainmentIn game design, creating game characters with personalized appearances enhances player immersion and personalized experience.
- Film and Animation ProductionIn film or animation production, it enables the rapid generation or modification of character designs, improving production efficiency and reducing costs.