3DTopia 2.0 - A large-scale 3D object generation model launched by Shanghai AI Lab in collaboration with Nanyang Technological University.

What is Scholar's Treasures 2.0?

3DTopia 2.0 is a large-scale 3D object generation model jointly developed by the Shanghai Artificial Intelligence Laboratory and Nanyang Technological University. The model employs the innovative primitive-based 3D representation method PrimX, which encodes shape, texture, and material information into a compact tensor format, enabling high-resolution geometric modeling. Based on the Diffusion Transformer framework, 3DTopia 2.0 supports the efficient generation of high-quality 3D assets with physically based rendering (PBR) characteristics from text or image input. The model code is open-source and offers a free commercial license, with the potential to revolutionize 3D content creation workflows in industries such as gaming, film and television, architecture, and design.

The main functions of Scholar's Treasures 2.0

• Multimodal input generates 3D objectsIt can quickly generate corresponding 3D models based on text descriptions or image inputs.
• Highly efficient generation processThe model can be converted from input to 3D model within five seconds, greatly improving creation efficiency.
• High quality and fine textureThe generated 3D objects have smooth geometry and spatially varying textures and materials, closely resembling the feel of real physical materials.
• Directly applied to game engines and design softwareThe generated 3D model can be used directly in game engines and industrial design software without additional processing.
• Supports high-resolution geometryBased on PrimX notation, it can model high-resolution 3D geometry.

The technical principles of Scholar's Essence 2.0

• PrimX notationA novel primitive-based 3D representation method encodes the shape, albedo, and material information of a 3D object into a compact tensor format. Each primitive is a small voxel, parameterized by its 3D position, global scaling factor, and corresponding spatially varying payload (including SDF, RGB, and material information).
• Original patch compressionA three-dimensional variational autoencoder (VAE) is used to compress the spatial information of each primitive to obtain latent primitive labels. The process employs 3D convolutional layers to compress the payload of primitives from a high-dimensional space to a low-dimensional latent space, providing efficient input for subsequent generative models.
• Latent Primitive DiffusionBased on the Diffusion Transformer (DiT) framework, the model learns how to progressively remove noise from random noise to generate latent primitive labels that meet the input conditions. The process simulates diffusion and denoising in physical processes, and can generate 3D objects with high-resolution geometry and PBR materials.
• Differentiable renderingPrimX notation supports differentiable rendering, allowing models to learn directly from 2D image data, thus improving the model's ability to learn from existing image resources.

The project address for "Scholar's Abundance 2.0"

• Github repository:https://github.com/3DTopia/3DTopia-XL
• arXiv technical paper:https://arxiv.org/pdf/2409.12957

Application Scenarios of Scholar's Abundance 2.0

• Game developmentIn game design, it can quickly generate various 3D game assets, such as characters, props, and environmental elements, improving the efficiency and richness of game development.
• Film and animation productionUsed to create 3D scenes and character models for movies or animations, reducing the time and cost of manual modeling while providing more creative freedom.
• Virtual Reality (VR) and Augmented Reality (AR)Generate realistic 3D environments and objects for virtual reality and augmented reality applications, enhancing the user experience.
• Architecture and Urban PlanningIn architectural design and urban planning, it can quickly generate 3D building models and urban landscapes to help designers and planners refine their designs and showcase their effects.