MagicClay - A 3D modeling tool from Adobe, providing text-guided local sculpting of 3D models.

What is MagicClay?

MagicClay is a 3D modeling tool from Adobe that combines mesh and directed distance field (SDF) technology, allowing artists to sculpt specific parts of a 3D model based on text prompts while keeping other areas of the model unchanged. MagicClay supports the generation of textured 3D models and enables non-destructive editing of local meshes, allowing artists to perform more intuitive and detailed editing of 3D models based on text prompts. MagicClay transforms text-to-image generation capabilities into a practical modeling tool used by artists in their iterative workflows.

MagicClay's main functions

• Text-guided 3D sculptingMagicClay supports user-input text prompts to guide the sculpting of specific areas of a 3D model, enabling localized editing.
• Hybrid representation: Combining the representations of meshes and directed distance fields (SDFs) to leverage the strengths of both for 3D modeling.
• Non-destructive editingUsers can edit selected areas without affecting other parts of the model, preserving the integrity of the original model.
• Sequential editingIt supports multiple edits of the same mesh, allowing artists to gradually refine the model.
• Preserve original attributesDuring the editing process, the original triangulation properties, such as vertex groups, are preserved for animation and other subsequent processing.
• High-resolution rendering: Efficiently render SDF based on mesh representation to achieve high-resolution volumetric rendering.

MagicClay's technical principles

• Hybrid Mesh-SDF RepresentationA hybrid representation is used, comprising triangular meshes and SDF. This hybrid representation supports intuitive control of the mesh and topological flexibility of the SDF.
• Consistency maintenanceDuring the optimization process, maintain the consistency between the mesh and SDF, and synchronize both using multi-view consistency loss and adaptive reconstruction.
• Differentiable renderingUsing differentiable rendering technology, render meshes and SDFs from different angles, requiring them to remain consistent in RGB rendering, opacity, and normal map.
• Localization and freezing lossBased on localization loss and freeze loss, it ensures that editing only occurs within the user-selected area, while unselected areas remain unchanged.
• Dynamic mesh reconstruction: Use differentiable mesh reconstruction techniques, such as ROAR, to dynamically update the mesh topology, including face splitting, edge folding, and edge flipping.
• Surface smoothing and regularizationThe explicit representation based on the grid defines a smoothing term, encodes local curvature variations using Laplacian vectors, and encourages grid smoothing.
• SDF Regularization: Encourage SDF to learn efficient distance field representations using Eikonal loss and other SDF-specific losses.

MagicClay's project address

• Project official website:amirbarda.github.io/MagicClay.github.io
• GitHub repository:https://github.com/amirbarda/MagicClay
• arXiv technical paper:https://arxiv.org/pdf/2403.02460

Application Scenarios of MagicClay

• 3D Art CreationArtists can quickly realize their creative ideas by using text prompts when creating 3D sculptures and models, without needing complex 3D modeling skills.
• Game developmentGame designers can quickly prototype and iterate on game characters, items, and environments to improve development efficiency.
• Animation ProductionIn animated film and television production, it involves creating and modifying complex 3D characters and scenes while preserving the vertex weights and rigging required for animation.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, design and optimize virtual objects and environments to provide a richer and more dynamic user experience.
• Education and trainingIn the field of education, it serves as a teaching tool to help students understand the basic concepts of 3D modeling and learn 3D design through practice.