FitDiT - A high-fidelity virtual try-on technology launched by Tencent and Fudan University

What is FitDiT?

FitDiT is a high-fidelity virtual try-on technology jointly developed by Tencent and Fudan University. Based on Diffusion Transformers (DiT), it focuses on high-resolution features to enhance the presentation of clothing details. FitDiT uses a clothing texture extractor and clothing prior evolution technology to enhance its ability to capture clothing textures such as stripes, patterns, and text. It also uses an expanded-relaxation masking strategy to optimize clothing size adaptation. FitDiT performs excellently in both qualitative and quantitative evaluation, quickly generating try-on images with realistic and complex details, and boasts fast inference speed, bringing a breakthrough to the field of virtual try-on.

FitDiT's main functions

• High-fidelity virtual try-onGenerate realistic try-on images, allowing users to see how they look in specific clothing in different scenarios.
• Texture-aware preservationBased on a garment texture extractor and garment prior evolution, it accurately captures and reproduces complex textures on garments, such as stripes, patterns, and text.
• Size-aware fittingThe expansion-relaxation masking strategy is used to adapt to the length and shape of different garments, preventing the leakage of garment shape information during cross-category fitting and achieving more accurate garment fitting.
• Rapid reasoningWhile maintaining a high-fidelity try-on effect, the DiT structure has been optimized, making the inference time for a single 1024×768 image only 4.57 seconds, thus improving the efficiency of the try-on process.

FitDiT's technical principles

• Diffusion Transformers (DiT)FitDiT is based on the DiT architecture, which enhances the ability to process clothing details by allocating more parameters and attention to high-resolution features.
• Clothing texture extractorA specialized garment texture extractor is introduced, which fine-tunes garment features based on prior evolution of garments to better capture the rich details of garments.
• Frequency domain learningBased on a customized frequency distance loss function, it enhances high-frequency clothing details and improves the fidelity of clothing texture and details.
• Expand-Relax Masking StrategyTo address the size-perceived fitting problem, an expansion-relaxation masking strategy is employed to adapt to the correct length of the garment, preventing the generation of garments that cover the entire mask area during cross-category fittings and improving fitting accuracy.
• Structural slimmingThe DiT structure is optimized by removing the text encoder, which has a smaller impact on virtual try-on, reducing the number of model parameters and improving the speed of model training and inference.
• Hybrid attention mechanismIn DenoisingDiT, a hybrid attention mechanism is used to inject clothing features extracted from GarmentDiT into the denoising process, thereby achieving the fusion of high-resolution features.

FitDiT's project address

• Project official website:byjiang.com/FitDiT
• GitHub repository:https://github.com/BoyuanJiang/FitDiT
• arXiv technical paper:https://arxiv.org/pdf/2411.10499

Application scenarios of FitDiT

• e-commerce platformClothing retail websites allow consumers to see how they look in different clothes while shopping online, enhancing their shopping experience and satisfaction.
• fashion industryDesigners showcase their designs, allowing customers to preview how the clothing will look on them before purchasing, thus increasing the appeal of the designs.
• Personalized customizationOur clothing customization service provides customers with a personalized fitting experience, ensuring that the size and style of the customized clothing perfectly meet their needs.
• Augmented Reality (AR) and Virtual Reality (VR)In AR and VR applications, a more realistic try-on experience is provided, allowing users to try on clothes and dress up their virtual avatars in a virtual environment.
• social mediaSocial media platforms allow users to try on different clothing styles when sharing photos or videos, increasing interactivity and entertainment.