DeepMesh - A 3D mesh generation framework developed by Tsinghua University and Nanyang Technological University.

What is DeepMesh?

DeepMesh is a 3D mesh generation framework proposed by researchers from Tsinghua University and Nanyang Technological University. It generates high-quality 3D meshes based on reinforcement learning and autoregressive transformers. It optimizes mesh generation through two key innovations: first, an efficient pre-training strategy combining novel tokenization algorithms and improved data processing; and second, the introduction of reinforcement learning (especially Direct Preference Optimization, DPO) to align the generated meshes with human preferences. DeepMesh can generate meshes with complex details and accurate topology based on point cloud and image conditions, outperforming existing methods in both accuracy and quality.

Main functions of DeepMesh

• High-quality 3D mesh generationDeepMesh can generate 3D meshes with rich detail and accurate topology, suitable for a variety of complex geometries.
• Point cloud conditional generationDeepMesh can generate corresponding 3D meshes based on input point cloud data, and is suitable for various scenarios ranging from sparse point clouds to dense point clouds.
• Image conditional generationDeepMesh supports image-based conditional generation, which can generate 3D meshes from input 2D images.

DeepMesh's technical principles

• Autoregressive TransformerDeepMesh employs an autoregressive transformer as its core architecture, incorporating self-attention and cross-attention layers. It progressively generates mesh faces, predicting vertices and faces based on conditional inputs (such as point clouds or images). For point cloud conditional generation tasks, DeepMesh combines a perceptron encoder to extract point cloud features and integrates them into the transformer model.
• Efficient pre-training strategiesDeepMesh introduces an improved tokenization algorithm that significantly shortens sequence length while preserving geometric details through locally aware face traversal and block index coordinate encoding. The framework employs improved data preparation and processing strategies, filtering low-quality mesh data and enhancing training efficiency through a truncation training strategy.
• Aligning reinforcement learning with human preferencesDeepMesh introduces Direct Preference Optimization (DPO), which collects preference pairs for reinforcement learning training through human evaluation and 3D metric-based scoring. This makes the generated meshes geometrically accurate and more aesthetically pleasing to humans.
• End-to-end differentiable mesh representationDeepMesh supports end-to-end differentiable mesh representations, allowing for dynamic topology changes. This differentiability enables the model to be optimized via gradient descent, further improving the quality of the generated mesh.

DeepMesh project address

• Project official website:https://zhaorw02.github.io/DeepMesh/
• Github repository:https://github.com/zhaorw02/DeepMesh
• arXiv technical paper:https://arxiv.org/pdf/2503.15265

Application scenarios of DeepMesh

• Virtual environment constructionDeepMesh can generate realistic 3D mesh models for building virtual scenes in virtual reality, such as virtual buildings and virtual cities.
• Dynamic content generationThrough reinforcement learning optimization, DeepMesh can dynamically generate 3D models based on real-time data in the game, enhancing the game's immersion and interactivity.
• Character animationDeepMesh can generate high-quality 3D character models and supports complex animation production needs, such as skeletal rigging and animation rendering.
• Dynamic medical simulationThrough reinforcement learning optimization, DeepMesh can generate dynamic medical models, such as cardiac motion simulations, to help doctors better understand organ movement and function.
• Product ModelingDeepMesh can be used to generate 3D models of industrial products, supporting complex design and manufacturing processes.