Prometheus - A 3D Perception Potential Diffusion Model Developed by Zhejiang University in Collaboration with Ant Financial and Other Universities

What is Prometheus?

Prometheus is an innovative 3D-aware latent diffusion model specifically designed for the rapid generation of text-to-3D scene content. It can generate 3D objects and scenes at the level of 3D within seconds, while maintaining high-quality output and good generalization ability. At its core, it leverages 2D prior knowledge to drive an efficient and generalizable 3D synthesis process. By representing 3D scene generation as a multi-view, feedforward, pixel-aligned 3D Gaussian generation process operating within the latent diffusion paradigm, Prometheus effectively generates 3D scenes with rich detail and accurate geometry from text descriptions. Fine-tuning based on a pre-trained text-to-image generation model and introducing an RGB-D latent space to decouple appearance and geometric information improves the fidelity and geometric quality of the generated content.

Main functions of Prometheus

• High-efficiency 3D generationIt can generate complex 3D scenes, including objects and the entire scene level, in seconds, improving the efficiency of 3D content creation.
• High-quality outputThe generated 3D scenes perform well in terms of visual fidelity and geometric quality, accurately reflecting the details and background information in the text description.
• Good generalization abilityBy training on large-scale single-view and multi-view datasets, Prometheus can generalize to a wide variety of 3D objects and scenes, with generalization capabilities comparable to Stable Diffusion.
• Multi-view consistencyThe generated 3D scene maintains consistency across multiple views and retains a stable visual effect even under large rotations or extreme perspectives.
• Text to 3D alignmentThe generated 3D scene can be accurately aligned with the input text prompts, ensuring that the generated content matches the user's description and expectations.

The technical principles of Prometheus

• Two-stage training framework
  • Phase 1: 3D Gaussian Variational Autoencoder (GS-VAE)A pre-trained image encoder (such as the Stable Diffusion encoder) encodes the RGB image and the predicted monocular depth map into a latent space. Cross-view information is integrated through a multi-view Transformer, and camera pose information is injected. The fused latent space variables are decoded into a pixel-aligned 3D Gaussian scene. The pixel-aligned 3D Gaussian scene serves as a scene-level representation.
  • Phase 2: Multi-View Potential Diffusion Model (MV-LDM)A denoising diffusion process is used to jointly predict the multi-view RGB-D latent space code, conditioned on camera pose and text cues. Starting from randomly sampled Gaussian noise, the multi-view latent space code is recovered through an iterative denoising process. The multi-view RGB-D latent space code is then used to generate the final 3D scene.
• The introduction of RGB-D latent space:Prometheus introduces an RGB-D latent space, decoupling appearance (RGB) and geometric information (D) to improve the fidelity and geometric quality of the generated models. The models can generate 3D Gaussians more efficiently while maintaining high-quality visuals.
• Feedforward generation strategy:Prometheus employs a feedforward generation strategy, which reduces generation time and improves generation efficiency compared to traditional optimization methods. By sampling multi-view RGB-D latent space codes from the latent space and decoding them into a 3D Gaussian scene using a GS-VAE decoder, it achieves fast and high-quality 3D scene generation.
• Classifier-free guidance (CFG):To ensure alignment between the generated 3D scene and the text prompts, Prometheus uses classifier-free guidance (CFG) to guide the multi-view generation process. By adjusting the guidance strength, it balances multi-view consistency and fidelity, avoiding inconsistencies in the generated results.
• Training on large datasets:Prometheus is trained on large-scale single-view and multi-view datasets to ensure good generalization ability. The datasets include various scene types, such as object-centric, indoor, outdoor, and driving scenarios, and text prompts are generated by a multimodal large language model.
• Loss function:During training, Prometheus uses a variety of loss functions, including mean squared error (MSE) loss, perceptual loss, and scale-invariant depth loss, to ensure that the generated 3D scene is visually and geometrically aligned with the input image and depth map.

Prometheus project address

• Project official website:https://freemty.github.io/project-prometheus
• arXiv technical paper:https://arxiv.org/pdf/2412.21117

Application scenarios of Prometheus

• Content creationQuickly generate realistic 3D scenes and objects for building virtual environments in VR and AR applications. For example, create virtual exhibitions, virtual tours, and virtual educational scenarios.
• Real-time interactionIn VR and AR applications, users can generate and modify 3D scenes in real time through text input, enhancing the interactive experience.
• Scene designIt can quickly generate various game scenes, such as cities, forests, and deserts, improving game development efficiency. Developers can generate complex 3D environments through simple text descriptions, reducing the time and cost of manual modeling.
• Concept DesignArchitects and interior designers can use Prometheus to quickly generate 3D models of buildings and interior designs for preliminary design and client presentations. By using text descriptions, 3D scenes of different styles and layouts can be generated rapidly, improving design efficiency.
• Virtual ShowcaseThe generated 3D scenes can be used for virtual demonstrations, allowing customers to immerse themselves in the design effects through VR devices and receive more intuitive feedback.