Edify 3D - NVIDIA's 3D generative modeling tool - AiBoss

What is Edify 3D?

Edify 3D is an advanced 3D asset generation solution from NVIDIA that can quickly synthesize high-quality 3D models from text prompts or reference images. Edify 3D can generate 3D assets with detailed geometry, clear topology, high-resolution textures, and PBR materials in just two minutes, supporting applications in video game design, extended reality, simulation, and many other fields. This technology combines a multi-view diffusion model and a Transformer-based reconstruction model, significantly improving the efficiency and scalability of 3D content production.

Edify 3D's main features

Text to 3D generationGenerates 3D assets with detailed geometry, clear topology, high-resolution textures, and PBR materials based on the input text description.
Image to 3D generationIt can automatically identify foreground objects from reference images and generate corresponding 3D assets.
Quick generationGenerate high-quality 3D assets in less than 2 minutes to meet the needs of rapid production.
High-resolution texturesSupports generating textures up to 4K resolution, ensuring the visual quality of generated assets.
PBR material supportThe generated 3D assets include physically based rendering (PBR) materials, which allow the assets to maintain a realistic appearance under different lighting conditions.
UV maps and material mapsThe generated assets include organized UV maps and material maps, which facilitates subsequent editing and rendering.

The technical principles of Edify 3D

Multi-view diffusion modelThis technique uses multi-view diffusion models to synthesize the RGB appearance and surface normal images of objects. These models can handle text cues and camera poses, synthesizing the appearance of objects from multiple viewpoints.
Transformer Basic Reconstruction ModelBased on synthesized multi-view RGB and surface normal images, a Transformer model is used to predict the geometry, texture, and material of 3D shapes. The model represents 3D objects as latent tokens, with mesh geometry derived from isosurface extraction and mesh processing.
Cross-view attention mechanismThe multi-view diffusion model extends the self-attention layer, allocates attention across different viewpoints, and improves viewpoint consistency.
High-resolution upsamplingUpsampling ControlNet improves the resolution of multi-view RGB images by conditionalizing the texture and surface normals of a given 3D mesh.
PBR renderingPhysically based rendering techniques are applied during the generation process to ensure the realism of materials and lighting effects.
Mesh post-processingThe generated 3D mesh undergoes post-processing, including retopology to quadrilateral meshes, UV mapping generation, and PBR attribute baking, to adapt to artistic and design needs.

Edify 3D project address

Project official website:research.nvidia.com/labs/dir/edify-3d
arXiv technical paper:https://arxiv.org/pdf/2411.07135

Application scenarios of Edify 3D

Video game designGame developers can quickly generate a 3D model of a medieval castle, including walls, towers, and internal structure, as well as corresponding textures and materials, for use in an upcoming strategy game.
Extended Reality (XR)Virtual reality experience designers create a realistic underwater world, including various marine life and coral reefs, to provide diving enthusiasts with an immersive virtual diving experience.
Film and video productionThe film production team generated a 3D scene of a futuristic city for a key chase scene in the science fiction film, which included skyscrapers, flying cars, and alien plants.
Simulation and trainingThe military training simulator generates complex urban environments and battlefield terrains for soldiers to conduct tactical training and simulate combat.
Architecture and Urban PlanningUrban planners quickly generate 3D city models based on the latest urban design drafts, including new residential areas, business centers, and parks, for public demonstration and feedback collection.