CatVTON - A virtual try-on technology jointly developed by Sun Yat-sen University and Pixocial

What is CatVTON?

CatVTON is an advanced virtual try-on technology jointly developed by Sun Yat-sen University and Pixocial. Based on a lightweight architecture and efficient training strategy, it achieves high-quality virtual try-on results. CatVTON is characterized by requiring only a minimal number of trainable parameters (approximately 49.57M), seamlessly transferring clothing onto the target person while maintaining consistency in detail. It abandons traditional complex network structures such as ReferenceNet and additional image encoders, simplifying the inference process and eliminating the need for preprocessing steps such as pose estimation, human anatomy, or text input. Trained on limited public datasets, CatVTON demonstrates excellent generalization ability in complex environments, bringing revolutionary changes to the fashion industry and consumer experience.

CatVTON's main functions

• Person-to-person clothing transferCatVTON transfers the clothing worn by one person onto another, achieving a personalized virtual try-on effect.
• Clothing to people trying onUsers can upload a flat lay image of clothing and a photo of a person, and the model will automatically fit the clothing onto the person.
• Multi-category supportIt supports trying on various clothing categories, including tops, pants, skirts, and suits.
• Consistency in detailsMaintain consistency in the shape, texture, and details of the garment in the fitting results.
• Simplify operating proceduresUsers do not need to perform complex preprocessing, such as pose estimation or human body analysis; they only need to provide simple image input.

CatVTON's technical principles

• Lightweight network architectureCatVTON is based on a lightweight network design, mainly including VAE and UNet, which reduces the number of model parameters and computational requirements.
• Efficient parameter trainingThe key training modules, such as the self-attention mechanism, were identified through experiments, and the modules were fine-tuned to achieve high-quality fitting results.
• Spatial Dimension StitchingDuring the input phase, the images of people and clothing are stitched together in a spatial dimension to ensure consistency between the two in the feature space.
• Simplify the reasoning processThis method omits traditional, complex preprocessing steps and allows for direct try-on using clothing reference images and target person images.
• Remove unnecessary conditionsIt does not rely on a text encoder and cross-attention mechanism, thus reducing the complexity of the model.

CatVTON project address

• Project official websitezheng-chong.github.io/CatVTON
• GitHub repository:https://github.com/Zheng-Chong/CatVTON
• HuggingFace model library:https://huggingface.co/zhengchong/CatVTON
• arXiv technical paper:https://arxiv.org/pdf/2407.15886v1

Application scenarios of CatVTON

• e-commerce platformOnline retailers are integrating CatVTON, allowing users to preview how clothing would look on them before purchasing, thus improving the shopping experience and satisfaction.
• Fashion DesignFashion designers use CatVTON to quickly preview the fit of their design sketches, accelerating the design and feedback process.
• Personalized recommendationsE-commerce platforms use CatVTON to provide users with personalized clothing recommendations, increasing user engagement and purchase conversion rates.
• social mediaUsers can create and share personalized virtual try-on photos on social media with CatVTON, increasing interactivity and entertainment.
• Augmented Reality (AR) ApplicationsIn AR virtual try-on applications, CatVTON provides a more realistic try-on experience, allowing users to try on clothes in a virtual environment.
• Virtual fashion showFashion brands are using CatVTON to showcase their clothing online, providing viewers with an immersive viewing experience.