Dive3D - A text-to-3D generation framework jointly developed by Peking University and Xiaohongshu.

What is Dive3D?

Dive3D is a text-to-3D generation framework developed in collaboration between Peking University and Xiaohongshu. The framework replaces the traditional KL divergence objective with a score-implicit matching (SIM) loss, effectively avoiding pattern collapse and significantly improving the diversity of generated 3D content. Dive3D excels in text alignment, human preference, and visual fidelity, achieving excellent quantitative results in the GPTEval3D benchmark, demonstrating its powerful ability to generate high-quality, diverse 3D assets.

Main functions of Dive3D

• Diverse 3D content generationIt generates 3D models with various styles and details based on text prompts, avoiding the pattern collapse problem common in traditional methods, i.e., the generated results are too monotonous and similar.
• High-quality 3D model generationIt supports generating 3D models with high visual fidelity, including fine texture details, realistic geometry, and reasonable lighting effects.
• Good text alignment capabilitiesIt supports generating 3D models that closely match the input text descriptions, accurately reflecting the various elements and features mentioned in the text.
• Supports multiple 3D representation formatsIt supports the generation of different types of 3D representations, such as Neural Radiation Fields (NeRF), Gaussian Splatting, and Mesh, to meet the needs of different application scenarios and users.

The technical principles of Dive3D

• Score Implicit Matching (SIM) LossOne of the core technologies of Dive3D is the loss function based on KL divergence (such as that used in Score Distillation Sampling, SDS). Traditional loss functions based on KL divergence (such as those used in Score Distillation Sampling, SDS) lead to pattern-seeking behavior, causing generative models to tend to generate samples from high-density regions, thus limiting the diversity of generation. SIM loss directly matches the probability density gradient field (score) of the generated content with the score of the diffusion prior, avoiding the problem of KL divergence and encouraging the model to explore multiple high-probability regions, thereby improving the diversity of generation while maintaining fidelity.
• Unified divergence perspective frameworkDive3D integrates diffusion distillation and reward-guided optimization into a divergence-based framework. This framework comprises three core divergence losses: Conditional Diffusion Prior Loss (CDP), Unconditional Diffusion Prior Loss (UDP), and Reward Loss (ER). By appropriately combining and adjusting the weights of the losses, a balance is achieved between generated diversity, text alignment, and visual quality.
• Optimization based on diffusion modelDive3D uses a pre-trained 2D diffusion model (such as Stable Diffusion) as prior knowledge to optimize the 3D representation into a 2D image that matches the text prompt based on multi-view rendering. During the optimization process, the 3D model is iteratively rendered, the loss is calculated, and the model parameters are updated so that the rendered image of the generated 3D model gradually approaches the image distribution expected by the pre-trained diffusion model.
• Highly efficient optimization algorithmsTo improve optimization efficiency, Dive3D employs efficient optimization algorithms and strategies. For example, it introduces Classifier-Free Guidance (CFG) technology to better balance the generation effects of text-conditional and unconditional methods during the optimization process; and it accelerates the optimization convergence speed and reduces generation time by reasonably setting parameters such as noise timetable and optimization step size.

Dive3D project address

• Project official websitehttps://ai4scientificimaging.org/dive3d/
• GitHub repositoryhttps://github.com/ai4imaging/dive3d
• arXiv technical paper: https://arxiv.org/pdf/2506.13594

Application scenarios of Dive3D

• Game developmentQuickly generate game characters, items, and scenes. Based on the descriptions in the game script, automatically generate game character models with different styles and details, reducing the workload of art design.
• Film and television productionProvides creative prototypes and concept designs for film, television, and animation production. Generates 3D models of scenes and characters based on script descriptions, helping directors and art directors to better develop creative ideas.
• Architectural DesignIt generates architectural models based on text descriptions, helping architects quickly showcase design concepts and compare and optimize different options.
• Virtual scene constructionGenerate realistic virtual scenes and objects for VR and AR applications. In virtual tourism applications, it generates corresponding 3D scenes based on the location description entered by the user, allowing the user to experience the virtual environment as if they were there.
• Science EducationGenerate complex scientific models, such as biological cells and molecular structures, to help students better understand abstract scientific concepts.