SHMT - A self-supervised makeup transfer technology launched by Alibaba DAMO Academy in collaboration with Wuhan University of Technology and other institutions.
SHMT (Self-supervised Hierarchical Makeup Transfer) is an advanced self-supervised makeup transfer technology jointly developed by Alibaba DAMO Academy and Wuhan University of Technology, among other institutions. The technology utilizes a latent diffusion model to achieve...
What is SHMT?
SHMT (Self-supervised Hierarchical Makeup Transfer) is an advanced self-supervised makeup transfer technology jointly developed by Alibaba DAMO Academy and Wuhan University of Technology, among other institutions. The technology utilizes a latent diffusion model, enabling the natural application of various makeup styles to target facial images without requiring paired training data. SHMT employs a "decoupling-reconstruction" strategy, combining a Laplacian pyramid and an iterative dual alignment module to achieve flexible control over texture details under different makeup styles and dynamic correction of alignment errors, significantly improving the effectiveness and fidelity of makeup transfer. Its main advantage lies in its ability to handle complex facial features and expression changes, providing high-quality transfer results.
SHMT's main functions
- Efficient migration:It maintains efficiency and high quality when handling multiple makeup styles, and can apply diverse makeup styles naturally and accurately to a given facial image.
- Dynamic alignment correction:By iterating the double alignment (IDA) module, the injection conditions are dynamically adjusted in each denoising step to correct alignment errors caused by domain gaps between content and makeup representation.
- Diverse applications:It is applicable to multiple fields such as image processing, style transfer, and computer vision, such as online makeup try-on on e-commerce platforms.
SHMT technical principles
- Self-supervised learning:SHMT employs a self-supervised strategy for model training, following a "decoupling-reconstruction" paradigm. It does not rely on poor-quality falsified reference data, thus avoiding the problem of misleading the model. Through decoupling and reconstruction, the model can learn effectively even without labeled data.
- Layered texture detail processing:This technology breaks down the makeup process into multiple layers, including base makeup, eye makeup, and lip makeup. Different makeup details are decomposed layer by layer using the Laplacian pyramid method and then selectively integrated into the content representation of the face, thus allowing for more flexible adaptation to various makeup styles.
- Dynamically correct alignment error:By iterating through the Individual Alignment Module (IDA), the makeup injection process in the diffusion model is dynamically adjusted to gradually correct the mismatch between facial content and makeup style. In each denoising step, IDA uses intermediate noise results to dynamically adjust the injection conditions and correct alignment errors.
SHMT project address
- Github repository:https://github.com/Snowfallingplum/SHMT
- arXiv technical paper:https://arxiv.org/pdf/2412.11058
SHMT application scenarios
- Social media beauty filters:SHMT can be applied to social media platforms to provide users with real-time beauty filter effects, allowing them to preview different makeup styles before posting photos. This increases user interactivity and fun, enhancing the platform's user experience.
- Virtual makeup try-on application:In the e-commerce sector, SHMT technology can be integrated into virtual makeup try-on applications, allowing customers to try on various cosmetics online and enhancing the shopping experience. Users can upload their own photos, select different makeup styles, and preview the effects in real time.
- Film and game character design:In the entertainment industry, SHMT can be used to quickly change or design makeup for characters in movies and games, improving production efficiency. This not only saves time and costs but also provides more creativity and flexibility for character design.
- Personalized advertising creation:The advertising industry can use SHMT technology to customize models' makeup based on the preferences of the target audience, making advertisements more attractive. Through personalized makeup design, advertisements can better attract the target audience and improve conversion rates.