DreamPolish - A text-to-3D generation model jointly developed by Zhipu AI, Tsinghua University, and Peking University.

What is DreamPolish?

DreamPolish, developed by Zhipu AI, Tsinghua University, and Peking University, is a text-to-3D generation model that improves the generation of fine geometry and high-quality textures for complex objects using a two-stage approach. The first stage refines the geometry using multiple neural representations, while a polishing stage improves surface details. The second stage uses domain score distillation to guide texture generation towards a target domain that combines realism and consistency, significantly enhancing texture quality. DreamPolish surpasses existing technologies in both geometry and texture, opening up new possibilities for 3D asset creation.

Main functions of DreamPolish

• Fine geometry generationGenerates 3D object geometry with complex details.
• High-quality texture generationThe model produces realistic textures, improving the visual quality of 3D models.
• Multi-stage geometry refinementImprove the surface details of the model based on progressive geometry construction and surface polishing.
• Domain Score Distillation (DSD)Introducing a new scoring distillation objective to balance texture realism and generation stability.
• Hybrid 3D generationBy combining the diffusion model of 2D images with 3D consistency constraints, the generation quality of 3D content can be improved.

The technical principles of DreamPolish

• Progressive geometry construction:
  • Starting with a rough 3D structure, the geometry is gradually refined using different neural representations (such as NeRF, NeuS, and DMTet).
  • Through iterative refinement, the model can generate complex geometries while maintaining computational efficiency.
• Surface polishingIn the final stage of geometry construction, the model is smoothed using pre-trained normal estimation to eliminate artifacts that may have been generated in the previous stage.
• Domain Score Distillation (DSD):
  • Based on the DSD target, the model is guided to a target domain that contains realistic and consistent rendering, improving texture quality.
  • By combining classifier-free guidance (CFG) and variational distribution guidance, a balance is struck between generative diversity and stability.
• Hybrid 3D generation:
  • High-quality textures from 2D images are transferred to 3D asset generation using a pre-trained 2D diffusion model and 3D consistency constraints.
  • Based on fractional distillation, the distributions of 2D and 3D representations are aligned, reducing discrepancies and artifacts.
• Balancing realism and stabilityBased on DSD technology, this approach addresses the challenge of maintaining training stability when generating realistic textures, avoiding oversaturation and other artifacts caused by excessively high CFG weights.

DreamPolish project address

• Project official website:deep-diver.github.io/ai-paper-reviewer/paper-reviews/2411.01602
• arXiv technical paper:https://arxiv.org/pdf/2411.01602
• Experience the demo online:https://huggingface.co/papers/2411.01602

Application scenarios of DreamPolish

• Virtual Reality (VR) and Augmented Reality (AR):In virtual reality and augmented reality applications, create realistic 3D environments and objects to enhance the user experience.
• Film and video production:Used in special effects and 3D scenes generated in movies to reduce the cost and complexity of actual shooting.
• Video game development:Game developers can quickly generate game assets with complex geometry and realistic textures, improving development efficiency.
• 3D printing:Text descriptions can be directly converted into 3D models for use in 3D printing, making personalized product customization more convenient.
• Education and training:Create educational content, such as 3D reconstructions of historical sites, to provide immersive learning experiences.