Fancy123 - 3D mesh generation technology launched by Huazhong University of Science and Technology and South China University of Technology

What is Fancy123?

Fancy123 is a 3D mesh generation technology developed by Huazhong University of Science and Technology and South China University of Technology. It generates high-quality 3D meshes from single images based on plug-and-play morphing techniques. The method includes two enhancement modules and a backprojection operation, addressing local inconsistencies in multi-view images, improving mesh fidelity to the input image, and ensuring high resolution, respectively. The appearance enhancement module aligns pixels based on morphed 2D multi-view images, while the fidelity enhancement module matches the input image with the morphed 3D mesh. Fancy123 demonstrates significant performance improvements in both qualitative and quantitative experiments and can be seamlessly integrated into existing single-image to 3D methods.

Fancy123's main functions

• Generating a 3D mesh from a single imageGenerates 3D mesh models with high visual appeal, color clarity, and input fidelity from a single RGB image.
• Solving local inconsistencies in multi-view imagesBased on the appearance enhancement module, inconsistencies in images generated by the 2D multi-view diffusion model are corrected, thereby improving multi-view consistency.
• Improve the fidelity of the grid for the input image.Based on the fidelity enhancement module, the 3D mesh is adjusted to more accurately match the input image, enhancing the similarity between the mesh and the input image.
• Ensure high definitionBased on the back projection operation, the input image and the deformed multi-view image are projected onto the mesh generated by LRM, which improves the clarity of the mesh color and removes the blur effect.

Fancy123's technical principles

• Multi-view diffusion modelGenerate a set of multi-view images based on the input image using a 2D multi-view diffusion model.
• Large Reconstruction Model (LRM): Quickly reconstruct an initial 3D mesh from a multi-view image using LRM.
• Appearance enhancement module: Optimizes mesh-based 2D deformable field alignment of misaligned pixels in multi-view images. The deformed multi-view image is back-projected onto the initial mesh, enhancing appearance quality and reducing ghosting effects.
• Fidelity enhancement moduleThe mesh is optimized based on parametric mesh deformation (using Jacobian fields) to make it closer to the input image. The camera parameters of the input image are estimated to maintain the correspondence with the input image during deformation.
• Back projection operationAfter 2D and 3D deformation, Fancy123 performs a backprojection operation, mapping the input image and the deformed multi-view image onto the mesh almost losslessly, improving the clarity of the mesh colors.
• Plug and play moduleFancy123’s two enhancement modules are designed to be plug-and-play, enabling them to work at inference time and supporting seamless integration into a variety of existing single-image to 3D methods.

Fancy123's project address

• Github repository:https://github.com/YuQiao0303/Fancy123
• arXiv technical paper:https://arxiv.org/pdf/2411.16185

Application Scenarios of Fancy123

• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it is used to create 3D models from existing 2D images, providing a more immersive experience.
• Game developmentGame designers can quickly generate 3D game assets, reducing the time and cost of manual modeling.
• Film and animation productionArtists create complex 3D characters and scenes for use in film and animation production.
• 3D printingConvert 2D designs or photos into 3D models for use in 3D printing various items, such as toys and artwork.
• e-commerceE-commerce platforms provide 3D displays for products, helping consumers understand them more intuitively.