ClotheDreamer - A 3D clothing generation technology developed by Shanghai University in collaboration with Tencent and other universities.

What is ClotheDreamer?

ClotheDreamer is a 3D clothing generation technology jointly developed by Shanghai University, Shanghai Jiao Tong University, Fudan University, and Tencent YouTu Lab. It can generate high-fidelity, wearable 3D clothing assets based on text descriptions. ClotheDreamer uses 3D Gaussian as its foundation, employing Disentangled Clothes Gaussian Splatting (DCGS) for optimized separation of clothing from the human body, and bidirectional Score Distillation Sampling (SDS) to enhance clothing rendering quality. ClotheDreamer supports custom clothing templates, allowing generated clothing to adapt to different body types, making it suitable for virtual try-on and physically accurate animation.

ClotheDreamer's main functions

• Text-driven 3D clothing generationAutomatically generate corresponding 3D clothing models based on text descriptions.
• High-fidelity renderingThe generated 3D clothing has a high degree of detail and realism.
• WearabilityThe generated clothing models are wearable and can be used for virtual try-on.
• Physically accurate animationSupports physically accurate animation effects for generated clothing.
• Custom template inputIt supports users uploading custom clothing templates to generate personalized 3D clothing.
• Adaptable to different body typesThe generated clothing can fit virtual characters of different body types.

ClotheDreamer's technical principles

• Disentangled Clothes Gaussian Splatting (DCGS)A novel representation method that represents clothing and the human body as Gaussian models, supporting independent optimization and rendering.
• Bidirectional Score Distillation Sampling (SDS)The pre-trained 2D diffusion model is used to optimize the rendering of 3D clothing and human body, providing guidance for the RGBD rendering of clothing and human body respectively, thereby improving the quality of the generated data.
• Text description parsingUse a language model (such as ChatGPT) to parse text descriptions, determine the type and characteristics of clothing, and provide a basis for the initialization of clothing models.
• Zero-shot learningGenerate corresponding 3D clothing models when a specific type of clothing is not seen.
• New pruning strategiesThe proposed trimming strategy for loose-fitting clothing avoids the erroneous removal of useful Gaussian points during the optimization process, thus maintaining the integrity of the clothing.
• Template-guided clothing generationUse custom clothing template meshes to guide the generation of 3D clothing, improving the personalization and practicality of the generation process.

ClotheDreamer's project address

• Project official website:ggxxii.github.io/clothedreamer
• GitHub repository:https://github.com/ggxxii/clothedreamer
• arXiv technical paper:https://arxiv.org/pdf/2406.16815

Application scenarios of ClotheDreamer

• Fashion DesignDesigners can quickly generate and iterate 3D clothing models, accelerating the design process and previewing the design effects in a virtual environment.
• Virtual try-onIn e-commerce, customers try on clothes in a virtual environment, improving the shopping experience and reducing return rates.
• Games and entertainmentGame developers design diverse outfits for game characters, providing richer and more personalized character customization options.
• Film and animation productionIn the film and animation industry, the rapid generation and animation of 3D costumes improves production efficiency and reduces costs.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, provide users with an immersive clothing try-on and design experience.