MagicTryOn - A video virtual try-on framework jointly developed by Zhejiang University, vivo, and other institutions.

What is MagicTryOn?

MagicTryOn is a video virtual try-on framework developed by the School of Computer Science and Technology at Zhejiang University and vivo Mobile Communications, among other institutions. Based on the video diffusion Transformer, the framework replaces the traditional U-Net architecture with the more expressive Diffusion Transformer (DiT), combining it with a fully self-attention mechanism to achieve spatiotemporal consistency modeling of videos. The framework employs a coarse-to-fine clothing preservation strategy, effectively preserving clothing details by integrating clothing tags during the embedding stage and introducing semantic, texture, and contour conditions during the denoising stage. MagicTryOn demonstrates performance surpassing state-of-the-art methods on both image and video try-on datasets, exhibiting excellent performance in evaluation metrics, visual quality, and generalization ability in outdoor scenes.

Main functions of MagicTryOn

• Clothing details preservedIt accurately simulates the texture, pattern, and outline of clothing, maintaining realism and stability during character movement.
• Spatiotemporal consistency modelingTo ensure the continuity between frames in the video, avoid clothing flickering and shaking, and provide a smooth try-on experience.
• Multi-condition guidanceBased on various conditions such as text, image features, clothing markings, and outline markings, it generates more realistic and detailed try-on effects.

The technical principles of MagicTryOn

• Diffusion Transformer (DiT) ArchitectureDiT's modular design allows for more flexible injection of conditional information, enabling multi-level clothing features to be effectively integrated into the denoising process. DiT's built-in fully self-attention mechanism can jointly model the spatiotemporal consistency of the video, capturing intra-frame local details and inter-frame dynamic changes.
• Clothing maintenance strategy from coarse to fine:
  • coarse strategyDuring the embedding phase, garment tokens are injected into the input token sequence, and the grid size of the Rotated Position Encoding (RoPE) is expanded so that garment tokens and input tokens can share a consistent position encoding.
  • Detailed strategyIn the denoising stage, semantically guided cross-attention (SGCA) and feature-guided cross-attention (FGCA) modules are introduced to provide fine-grained guidance on clothing details. SGCA uses text tags and CLIP image tags as input to provide a global semantic representation of the clothing; FGCA combines clothing tags and outline tags to further inject detailed local information.
• Mask-Aware LossBy introducing mask-aware loss, the model can focus more on generating clothing areas, improving the detail fidelity of clothing areas and the realism of the overall synthetic result.
• Spatiotemporal consistency modelingThis method uses a fully self-attention mechanism to jointly model the spatiotemporal consistency of videos, avoiding the limitations of traditional methods that model spatial and temporal information separately.

MagicTryOn project address

• Project official website:https://vivocameraresearch.github.io/magictryon/
• GitHub repository:https://github.com/vivoCameraResearch/Magic-TryOn/
• arXiv technical paper:https://arxiv.org/pdf/2505.21325

Application scenarios of MagicTryOn

• Online shoppingUsers can try on different clothes online, enhancing their shopping experience.
• Fashion DesignDesigners can quickly preview the appearance of clothing, accelerating the design process.
• Virtual fitting roomProvide virtual fitting services for physical stores to reduce the use of physical fitting rooms.
• Advertising and MarketingBrands create personalized try-on ads to attract consumers.
• Games and EntertainmentTry on virtual clothes in real time during the game to enhance immersion.