ConsistentDreamer - Huawei's technology for generating 3D assets from a single image.

What is ConsistentDreamer?

ConsistentDreamer, a novel image-to-3D asset generation technology proposed by Huawei's Munich Research Center, generates view-consistent 3D meshes from a single image. The method addresses the shortcomings of traditional methods in achieving multi-view consistency through Gaussian optimization guided by multi-view prior images. ConsistentDreamer first generates a set of multi-view images with fixed perspectives, then optimizes the coarse shape of the 3D model based on fractional distillation sampling (SDS) loss and a diffusion model. It refines the surface by balancing the optimization of coarse shape and fine details through dynamic task weights, and by introducing opacity, depth distortion, and normal alignment losses.

Main functions of ConsistentDreamer

• 3D Consistency EnhancementBy introducing 3D consistent structured noise and self-supervised consistency training, ConsistentDreamer can maintain highly consistent editing results across different views, solving the inconsistency problem of traditional 2D diffusion models in multi-view generation.
• High-resolution texture generationThe framework can generate edited results with fine textures and high definition, and performs well in complex scenes such as large-scale indoor scenes in ScanNet++.
• Complex pattern editing capabilitiesConsistentDreamer is the first method that can successfully edit complex patterns (such as grid or square patterns).
• Multiview Context InputBy using the surrounding view as input, ConsistentDreamer provides rich contextual information to 2D diffusion models, enhancing the models' 3D perception capabilities.
• Parallel editing workflowConsistentDreamer employs multi-GPU parallel processing and achieves efficient scene editing by separating NeRF fitting and diffusion model generation.
• Command-guided scene editingThe framework supports editing 3D scenes based on natural language instructions, generating high-quality results that are highly consistent with the instructions.

The technical principle of ConsistentDreamer

• Multi-view prior image guidanceConsistentDreamer first generates a set of fixed-view prior images from a single input image based on a multi-view generative model. These images serve as references during the optimization process, providing rich contextual information for the generation of the 3D model.
• Fractional distillation sampling (SDS)The coarse shape of the 3D model is optimized using fractional distillation sampling (SDS) loss. Specifically, random views are generated based on a pre-trained diffusion model (such as Zero-1-to-3), and consistency between views is ensured by selecting the prior image that is closest to the target view as a condition.
• Dynamic task weight balancingTo balance coarse shape optimization and fine detail optimization, ConsistentDreamer introduces dynamic task weights based on homoscedasticity uncertainty. These weights are automatically updated in each iteration, ensuring the stability and efficiency of the optimization process.
• Opacity, depth distortion, and normal alignment lossTo improve the quality of mesh extraction, ConsistentDreamer introduces opacity loss, depth distortion loss, and normal alignment loss. These help refine the surface, ensuring that the generated 3D mesh has a sharp surface and high-quality texture.
• Multi-view context input and consistency trainingConsistentDreamer takes the surrounding view as input, providing rich 3D contextual information to the diffusion model, and further enhances 3D perception capabilities through self-supervised consistency training.

ConsistentDreamer's project address

• arXiv technical paper:https://arxiv.org/pdf/2502.09278

Application scenarios of ConsistentDreamer

• High-fidelity editing of complex scenesConsistentDreamer is suitable for complex, large-scale indoor scenes (such as those in the ScanNet++ dataset), and can generate edited results with fine textures and high definition.
• Diverse style conversionIt supports various style conversion tasks, such as converting scenes to a specific art style (such as Van Gogh or Munch style), while preserving the details and textures of the original scene.
• Object-specific editingConsistentDreamer allows you to edit specific objects in a scene, such as changing a character's facial expression or the color of an object.
• Cross-view and cross-batch consistency: By introducing structured noise and self-supervised consistency training, ConsistentDreamer can maintain consistency across different views and batches.