Fashion-VDM - A virtual try-on technology jointly launched by Google and the University of Washington

What is Fashion-VDM?

Fashion-VDM is a virtual try-on technology jointly developed by Google and the University of Washington, based on a video diffusion model (VDM). Given an image of clothing and a video of a person, it can generate high-quality try-on videos of that person wearing specified clothing, preserving the person's identity and movements. Fashion-VDM addresses the issues of clothing detail and temporal consistency in video virtual try-on by using a diffusion model architecture, a freely guided segmentation classifier, and a progressive temporal training strategy. Even with limited video data, it improves clothing fidelity through joint image-video training, reaching a new level in the field of video virtual try-on.

The main functions of Fashion-VDM

• Generate virtual try-on videosFashion-VDM combines a given image of clothing with a video of a person to generate a video of that person wearing the clothing, preserving the person's original identity and actions.
• Maintain consistency in clothing details and time.This addresses the issues of missing clothing details and insufficient time consistency in existing video virtual try-on methods.
• Improve the fidelity of clothingBased on the Split Classifier-Free Guidance technology, it enhances the control and fidelity of clothing image details.
• Optimize video generation efficiencyA progressive time training strategy is used to generate 512px videos in a single pass of 64 frames, improving the efficiency and quality of video generation.
• Joint image-video trainingCombining image and video data during training, especially when video data is limited, can improve model performance.

The technical principle of Fashion-VDM

• Diffusion Model ArchitectureFashion-VDM is based on a diffusion model that gradually recovers a clear image or video from noise.
• 3D Convolution and Temporal Attention BlocksAdd 3D convolutions and temporal attention blocks to the main UNet network to maintain temporal consistency between video frames.
• Split Classifier-Free GuidanceA technique for independently controlling multiple conditional signals, enabling models to more precisely control the generated results and improve clothing fidelity and consistency between video frames.
• Progressive Time TrainingThe model trains through multiple stages, gradually increasing the length of video frames. Starting with image data, it gradually introduces longer video frames for training until it reaches the target of 64 frames.
• Joint image-video trainingDuring training, image and video data are used, and conditional network branching techniques are employed to skip updates to time blocks during image batch training, thereby improving data diversity and training stability.
• Preprocessing and encodingThe input video and clothing images are preprocessed to extract information such as human pose, clothing-independent frames, clothing segmentation, and clothing pose, and then encoded using different UNet encoders.

Fashion-VDM project address

• Project official website:johannakarras.github.io/Fashion-VDM
• arXiv technical paper:https://arxiv.org/pdf/2411.00225

Application Scenarios of Fashion-VDM

• Online clothing shoppingConsumers can use the virtual try-on function to see the effect of clothing more intuitively before purchasing, reducing the problem of returns and exchanges due to unsuitable size or style.
• Fashion Design and MarketingFashion designers and brands showcase their designs, anticipate market reactions, and reduce design and marketing costs.
• e-commerce platformE-commerce platforms can integrate Fashion-VDM technology to enhance the interactivity and fun of online shopping, thereby increasing consumers' willingness to buy.
• Virtual Fashion ShowVirtual fashion shows showcase clothing, providing a more exciting and realistic display effect.
• Social media marketingBrands and influencers can create engaging content on social media to increase user engagement and brand exposure.