MeshPad - A sketch-driven AI 3D mesh generation and editing tool

What is MeshPad?

MeshPad is an interactive, artistic mesh generation and editing tool based on sketch input. It can quickly convert simple 2D sketches into high-quality 3D mesh models, supporting real-time editing. Users can add or delete lines on the sketch to modify the 3D mesh, such as deleting regions or adding new geometry. It uses triangle sequence representation and captures complex geometric relationships through a Transformer model to generate high-quality meshes. A speculative prediction strategy significantly reduces computational costs and accelerates the editing process, with each edit taking only a few seconds. The generated mesh quality is 22% higher than existing methods at the Chamfer distance.

MeshPad's main functions

• Sketch-driven 3D mesh generationUsers only need to draw simple 2D sketches, and MeshPad can automatically convert these sketches into high-quality 3D mesh models.
• Real-time interactive editingMeshPad allows users to edit the generated 3D mesh in real time. Users can add, delete, or modify lines on the sketch, and the changes will be immediately reflected in the 3D mesh model.
• High-efficiency mesh processingMeshPad uses a triangular sequence representation, enabling efficient handling of mesh addition and deletion operations. Its built-in Transformer model captures complex geometric relationships, generating high-quality meshes.
• High-quality mesh outputMeshPad generates meshes that improve upon existing methods by 22% in terms of Chamfer distance, resulting in more accurate and detailed 3D models.

MeshPad's technical principles

• Triangle sequence representationMeshPad uses a triangular sequence representation to efficiently handle mesh addition and deletion operations.
• Transformer modelMeshPad uses a large Transformer model to capture complex geometric relationships. When handling mesh addition and deletion operations, the Transformer model can recursively generate new triangular meshes and seamlessly integrate the newly generated meshes with the existing model.
  • Add operationTransformer uses an autoregressive model to predict the position of new triangular patches and aligns vertex coordinates through "speculation prediction" to ensure that the new part blends perfectly with the original mesh.
  • Delete operation (Del)By analyzing vertices in the mesh through a classification network, "deletion labels" are provided to accurately remove unwanted parts without damaging other structures.
• speculative forecasting strategiesMeshPad introduces a speculative prediction strategy for vertex alignment. By predicting multiple output tags for vertices, it significantly reduces computational costs and accelerates the editing process. Each edit takes only a few seconds, achieving a truly real-time interactive experience.
• Real-time interaction and optimizationMeshPad's interactive design allows users to edit 3D mesh models in real time through simple sketching operations such as adding or deleting lines.

MeshPad project address

• Project official website:https://derkleineli.github.io/meshpad/
• arXiv technical paper:https://arxiv.org/pdf/2503.01425

Application scenarios of MeshPad

• Art and DesignQuickly convert sketches into 3D sculptures, animations, or game models.
• Architectural DesignIt converts hand-drawn sketches into architectural models and allows for real-time adjustments to design details.
• Industrial DesignGenerate and modify product prototypes to accelerate the design iteration process.