FabricDiffusion - A high-fidelity 3D clothing generation technology developed by Google in collaboration with Carnegie Mellon University.
FabricDiffusion is a high-fidelity 3D clothing generation technology jointly developed by Google and Carnegie Mellon University. It can transfer the textures and prints of real-world 2D clothing images to 3D clothing models of arbitrary shapes with high quality.
What is FabricDiffusion?
FabricDiffusion, a high-fidelity 3D clothing generation technology jointly developed by Google and Carnegie Mellon University, can transfer the textures and prints of real-world 2D clothing images to 3D clothing models of arbitrary shapes with high quality. FabricDiffusion corrects distortion in input texture images based on a denoising diffusion model and large-scale synthetic datasets, generating various texture maps including diffuse, roughness, normal, and metallicity. It achieves accurate relighting and rendering of 3D clothing under different lighting conditions, demonstrating excellent performance and generalization capabilities.
Main functions of FabricDiffusion
- High-quality texture transferAutomatically extracts and transfers the textures and prints from 2D clothing images to 3D clothing models.
- Processing multiple texturesIt can handle various types of textures, patterns, and materials.
- Generate multiple texture mapsIt can generate diffuse maps, as well as roughness, normal, and metallic texture maps.
- Rendering across lighting conditionsSupports accurate relighting and rendering of 3D clothing under different lighting conditions.
- Zero-shot generalizationGeneralize to real-world images when trained entirely using synthetically rendered images.
The technical principles of FabricDiffusion
- Denoising diffusion modelThe denoising diffusion model is used to learn how to recover distortion-free, tiling-compatible texture materials from distorted input texture images.
- Large-scale synthetic datasetsWe constructed a large-scale synthetic dataset containing over 100k textile color images, 3.8k material PBR texture maps, 7k prints, and 22 3D clothing meshes to train the model.
- Texture image correctionBased on model training, it corrects distortions in the input texture image and generates a flat texture map that is closely integrated with the physically based rendering (PBR) material generation process.
- Feature transfer: Transfer various features, including texture patterns, material properties, and detailed prints and logos, from a single garment image.
- Normalization and TileabilityThe generated texture map is normalized and tiled in the UV space of the garment, seamlessly integrating with existing PBR material estimation workflows.
- Conditional generationThe model generates corresponding textures based on the input clothing image conditions, achieving high-quality texture transfer from 2D to 3D.
FabricDiffusion project address
- Project official website:humansensinglab.github.io/fabric-diffusion
- arXiv technical paper:https://arxiv.org/pdf/2410.01801
Application scenarios of FabricDiffusion
- Virtual try-onIn e-commerce and fashion retail, creating virtual fitting rooms allows consumers to try on 3D clothing online, enhancing the shopping experience.
- Games and entertainmentIn game development, it enables the rapid generation of 3D clothing with realistic textures, enhancing the visual realism of game characters.
- Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, create realistic virtual environments and characters to enhance user immersion.
- Film and television productionIn film and television production, it generates or modifies clothing textures to improve the efficiency of special effects and costume design.
- Fashion design and prototypingDesigners explore new designs and textures, quickly create clothing prototypes, and accelerate design iteration.