AB
AB
AiBoss

All Categories

project

Direct3D-S2 - A high-resolution 3D generation framework jointly developed by Nanjing University, Fudan University, and other universities.

Direct3D-S2 is a high-resolution 3D generation framework jointly developed by researchers from Nanjing University, DreamTech, Fudan University, and Oxford University. It is based on sparse volume representation and an innovative spatial sparse attention (SSA) mechanism...

What is Direct3D-S2?

Direct3D-S2 is a high-resolution 3D generation framework jointly developed by researchers from Nanjing University, DreamTech, Fudan University, and Oxford University. Based on sparse volumetric representation and an innovative Spatial Sparse Attention (SSA) mechanism, it significantly improves the computational efficiency of the Diffusion Transformer (DiT) and substantially reduces training costs. The framework includes a fully end-to-end sparse SDF variational autoencoder (SS-VAE) using a symmetric encoder-decoder structure, supports multi-resolution training, and can be trained on up to 1024³ resolution using 8 GPUs. Direct3D-S2 surpasses existing methods in both generation quality and efficiency, providing powerful technical support for high-resolution 3D content creation.

Main functions of Direct3D-S2

  • High-resolution 3D shape generationGenerates high-resolution 3D shapes from images, supporting resolutions up to 1024³, with generated 3D models featuring fine geometric details and high visual quality.
  • Efficient training and reasoningSignificantly improves the computational efficiency of the Diffusion Transformer (DiT) and reduces training costs. Only 8 GPUs are required for training at 1024³ resolution.
  • 3D generation of image conditionsSupports image-based conditional generation, generating a corresponding 3D model based on the input image.

The technical principles of Direct3D-S2

  • Spatial Sparse Attention (SSA) MechanismThe input tokens are divided into blocks based on 3D coordinates. Block-level global information is extracted using sparse 3D convolution and pooling operations, reducing the number of tokens and improving computational efficiency. Based on the attention score from the compression module, important blocks are selected for fine-grained feature extraction, further optimizing the use of computational resources. Local features are injected using local window operations to enhance local feature interaction and improve generation quality. The outputs of the three modules are aggregated based on the predicted gating scores to generate the final attention result.
  • Sparse SDF Variational Autoencoder (SS-VAE)This method combines sparse 3D convolutional networks and Transformer networks to encode high-resolution sparse SDF volumes into sparse latent representations, and then reconstructs the SDF volumes based on the decoder. During training, SDF volumes at different resolutions are randomly sampled to improve the model's adaptability to data of varying resolutions, thereby enhancing training efficiency and generalization ability.
  • Image-Conditioned Diffusion Transformer (SS-DiT)Extracting sparse foreground tokens from the input image reduces interference from background tokens and improves the consistency between the generated 3D model and the input image. Based on Conditional Flow Matching (CFM) training, the model predicts the velocity field from noisy samples to the data distribution, achieving efficient 3D shape generation.

Direct3D-S2 project address

Application scenarios of Direct3D-S2

  • Virtual Reality (VR) and Augmented Reality (AR): Build realistic 3D environments, create personalized 3D characters, and integrate real-world scenarios for education and cultural heritage preservation.
  • Game developmentIt can quickly generate high-quality 3D game assets, achieve real-time 3D content generation, and generate customized content based on player input.
  • Product design and prototypingQuickly generate 3D models of products, virtually showcase products, and meet personalized design needs.
  • Film and animation productionGenerate high-quality 3D animated characters, create virtual scenes, and produce complex 3D special effects.
  • Education and trainingCreate virtual laboratories, generate 3D teaching models, and conduct virtual vocational training.
Back