AB
AB
AiBoss

All Categories

project

TRELLIS - A high-quality 3D generative model jointly developed by Microsoft, Tsinghua University, and the University of Science and Technology of China.

TRELLIS is a 3D generative model developed by Tsinghua University, the University of Science and Technology of China, and Microsoft Research. Based on the Structured LATent (SLAT) representation, it generates high-quality, diverse 3D assets from text or image cues...

What is TRELLIS?

TRELLIS is a 3D generative model developed by Tsinghua University, the University of Science and Technology of China, and Microsoft Research. Based on Structured LATent (SLAT) representation, it generates high-quality, diverse 3D assets from text or image cues. The model integrates sparse 3D mesh structures with dense visual features extracted from multiple perspectives, comprehensively capturing the geometric and appearance information of 3D assets. TRELLIS uses a modified flow transformer as its core to handle the sparsity of SLAT and is trained on a large-scale 3D asset dataset with up to 2 billion parameters. TRELLIS can generate richly detailed 3D models, supports multiple output formats, and allows for flexible editing of 3D assets.

TRELLIS main functions

  • High-quality 3D generationGenerate 3D assets with complex geometry and detailed textures based on text or image prompts.
  • Multi-format outputThe model supports generating 3D assets in multiple formats, including Radiance Fields, 3D Gaussians, and meshes.
  • Flexible editingIt allows users to partially edit the generated 3D assets, such as adding, deleting, or replacing specific areas, without needing to adjust the entire asset.
  • No fitting training requiredDuring training, there is no need to fit 3D assets, which simplifies the training process and improves efficiency.

TRELLIS's technical principles

  • Structured LATent (SLAT) meansUsing SLAT as a unified 3D latent representation, based on defining local latent variables on a sparse 3D mesh to encode the geometric and appearance information of 3D assets.
  • Multi-view visual featuresBased on a powerful visual foundation model, multi-view visual features are extracted and used to encode the structural and appearance information of 3D assets in detail.
  • Modified Stream ConverterBased on the modified stream transformer as the core of its 3D generation model, the transformer is particularly well-suited to handling the sparsity of SLAT and can efficiently generate 3D assets.
  • Two-stage generation processFirst, a sparse structure for SLAT is generated, and then local latent vectors are generated on the generated structure. This process allows the model to flexibly generate 3D representations in different formats.
  • Training and optimizationThe encoder and decoder are trained on a large-scale 3D asset dataset and optimized with specific reconstruction loss and KL penalty to ensure that the generated 3D assets are highly consistent with the original data.

TRELLIS project address

TRELLIS limitations

  • Step-by-step generation processThe generation process involves two main stages, which may result in lower generation efficiency compared to methods that generate the entire 3D model in one go.
  • Lighting and rendering limitationsThe model failed to separate the lighting effects in the image from the 3D assets when processing image cues, which may affect the accuracy and realism of the materials in physically based rendering of the 3D assets.
  • Future improvement directionsThe paper proposes areas for future research, including improving lighting processing and enhancing the model's ability to predict physically based rendering materials.

TRELLIS Application Scenarios

  • Game developmentIn game design, it enables the rapid generation of complex 3D models, such as characters, props, and environments, thereby improving the efficiency of game development.
  • Film and animation productionCreate 3D scenes and characters for movies or animations, reducing the time and cost of manual modeling.
  • Virtual Reality (VR) and Augmented Reality (AR)Generate realistic 3D content for VR and AR applications, enhancing user immersion and interactive experience.
  • Architecture and Urban PlanningGenerate architectural models and urban landscapes to assist architects and planners in design and simulation.
  • Education and trainingIn the field of education, various 3D models are generated for use in teaching and practical operations, such as medical visualization and engineering instruction.
Back