DiffSplat - A 3D generative framework jointly developed by Peking University and ByteDance.
DiffSplat is a novel 3D generation method that rapidly generates 3D Gaussian splats from text prompts and single-view images. It leverages a pre-trained text-to-image diffusion model, based on a powerful 2D prior...
What is DiffSplat?
DiffSplat is a novel 3D generation method that rapidly generates 3D Gaussian splats from text prompts and single-view images. By fine-tuning a pre-trained text-to-image diffusion model, it leverages strong 2D prior knowledge and introduces a 3D rendering loss to ensure consistency of generated 3D content across multiple views. DiffSplat's core advantages lie in its efficiency and flexibility, generating high-quality 3D objects within 1-2 seconds, supporting text-based, image-based, or combined inputs. A lightweight reconstruction model is used to construct structured Gaussian representations, providing high-quality data support for training.
DiffSplat's main functions
- Generate 3D Gaussian raster from text or imageDiffSplat can generate 3D Gaussian splats directly from text prompts or single-view images, maintaining 3D consistency.
- Efficiently utilize 2D prior knowledgeBy fine-tuning a large-scale text-to-image diffusion model, DiffSplat effectively leverages network-scale 2D prior knowledge while introducing 3D rendering loss to ensure 3D consistency of generated content across any view.
- Supports multiple input conditionsDiffSplat supports input of text conditions, image conditions, or a combination of both, allowing users to select the appropriate conditions as needed.
- Controllable generation capabilityDiffSplat can be combined with technologies such as ControlNet to achieve controllable 3D generation based on text prompts and various formats (such as normal maps, depth maps, and Canny edge maps).
DiffSplat's technical principles
- Based on a large-scale pre-trained text-to-image diffusion modelDiffSplat generates 3D Gaussian splats directly by fine-tuning these models, effectively leveraging 2D prior knowledge at the network scale.
- Lightweight reconstruction modelTo enable training, DiffSplat proposes a lightweight reconstruction model that can quickly generate multi-view Gaussian grids for scalable dataset construction.
- 3D rendering lossDiffSplat introduces 3D rendering loss to ensure that the generated 3D content remains consistent across any view.
DiffSplat's project address
- Project official website:https://chenguolin.github.io/projects/DiffSplat/
- GitHub repository:https://github.com/chenguolin/DiffSplat
- arXiv technical paper:https://arxiv.org/pdf/2501.16764
Application scenarios of DiffSplat
- 3D content creationSuitable for rapid prototyping and content creation. Designers can use DiffSplat to quickly generate initial versions of 3D models for proof-of-concept or further fine-tuning.
- Text to 3D generationDiffSplat excels in text-based 3D generation tasks, generating matching 3D models based on detailed text descriptions.
- Image to 3D ReconstructionDiffSplat supports generating 3D models from single images, accurately reflecting the shape and texture of the input image. It allows for the rapid generation of 3D models from existing image resources, useful in fields such as film and television special effects and game development.
- Downstream application supportThe results generated by DiffSplat can be directly used in a variety of downstream applications, such as 3D printing, virtual reality (VR), and augmented reality (AR).