LATTE3D - A model from NVIDIA for quickly generating 3D objects from text.
LATTE3D is a text-to-3D object generation model developed by NVIDIA researchers. It can quickly generate high-quality 3D content from text prompts (in just 400 milliseconds). The core of this technology lies in a technique called "...
What is LATTE3D?
LATTE3D is a text-to-3D object generation model developed by researchers at NVIDIA's Toronto AI Lab. It can quickly generate high-quality 3D content from text prompts (in just 400 milliseconds). The core of this technology lies in a method called "amortized optimization," which simultaneously optimizes a shared text conditional model across a large number of text prompts. This improves the model's generalization ability to new prompts, thereby reducing the time required to generate each 3D object.
LATTE3D official website entrance
- Official project homepage:https://research.nvidia.com/labs/toronto-ai/LATTE3D/
- Research Papers:https://drive.google.com/file/d/1HZ7EY1jFguiwxxetgQkpljrj0cxbhZXZ/view
Main functions of LATTE3D
- Text-to-3D compositing: It can generate corresponding 3D models based on text descriptions. Users can generate 3D objects with specific features and styles by inputting text prompts, such as "a plush amoeba crab wearing a top hat".
- Quick generation: LATTE3D can generate 3D objects in about 400 milliseconds, which allows it to respond to user input in real time and provide users with instant visual feedback.
- High-quality rendering: By combining neural fields and textured surface generation, LATTE3D is able to produce textured meshes with a high level of detail, providing visually convincing 3D rendering results.
- 3D stylization: LATTE3D can also be used as a 3D stylization tool, allowing users to apply new styles or themes to existing 3D assets, thereby creating diverse visual expressions.
LATTE3D's architectural approach
Training LATTE3D consists of two phases: First, volumetric rendering is used to train textures and geometry. To enhance the robustness of the cues, the training objective includes SDS gradients from 3D-aware image priors and a regularized loss that compares a mask of the predicted shape with 3D assets in the library. Then, surface-based rendering is used and textures are trained only to improve quality. Both phases use amortized optimization on a set of cues to maintain fast generation.
LATTE3D's method uses two networks: a texture network T and a geometry network G, both composed of a combination of triplanes and a U-Net. In the first stage, the encoders of both networks share the same weight set. In the second stage, the geometry network G is frozen and the texture network T is updated, and the triplanes are further upsampled using an MLP that embeds the input text.