Amodal3R - A conditional 3D generative model developed by Nanyang Technological University in collaboration with Oxford University and others.

What is Amodal3R?

Amodal3R is a conditional 3D generative model that infers and reconstructs the complete 3D shape and appearance from partially visible 2D object images. Built upon the "basic" 3D generative model TRELLIS, it utilizes occlusion prior knowledge to guide the reconstruction process by introducing a mask-weighted multi-head cross-attention mechanism and an occlusion-aware attention layer. Trained solely on synthetic data, Amodal3R performs exceptionally well in real-world scenarios, significantly outperforming existing two-step methods of "2D prediction completion + 3D reconstruction," setting a new benchmark for 3D reconstruction in occluded scenes.

Main functions of Amodal3R

• Occlusion-aware 3D ReconstructionFor 2D images with severe occlusion, Amodal3R can combine 2D fragment information with semantic inference to generate a complete 3D model.
• Beyond existing methodsCompared to the two-step method of "2D prediction completion + 3D reconstruction", Amodal3R performs better under occlusion conditions, establishing a new benchmark for 3D reconstruction.

Amodal3R's technical principles

• Basic 3D Generative Model ExtensionAmodal3R starts from a "basic" 3D generative model and extends it to handle occluded 2D images to recover reasonable 3D geometry and appearance.
• Mask-weighted multi-head cross-attention mechanismThe model introduces a mask-weighted multi-head cross-attention mechanism, which can better handle occlusion problems. Specifically, the attention mechanism is guided by a mask, making the model pay more attention to the visible parts during the generation process, and using prior knowledge of occlusion to infer the shape and texture of the occluded areas.
• Occlusion of the perception attention layerFollowing the mask-weighted multi-head cross-attention mechanism, Amodal3R introduces an occlusion-aware attention layer.
• Feature extraction based on DINOv2Amodal3R utilizes DINOv2 for high-quality visual feature extraction. The features provided by DINOv2 offer more contextual information for 3D reconstruction, helping the model to perform 3D reconstruction more accurately.
• Synthetic data training and generalization capabilitiesAmodal3R, trained solely on synthetic data, learns to reconstruct complete 3D objects even in real-world scenes with occlusion. This demonstrates the model's strong generalization ability, enabling it to apply knowledge learned from synthetic data to real-world scenarios.

Amodal3R's project address

• Project official website:https://sm0kywu.github.io/Amodal3R/
• HuggingFace model library:https://huggingface.co/Sm0kyWu/Amodal3R
• arXiv technical paper:https://arxiv.org/pdf/2503.13439

Application scenarios of Amodal3R

• Augmented Reality (AR) and Virtual Reality (VR)In AR and VR applications, Amodal3R can help reconstruct complete 3D models from partially visible 2D images, providing a more immersive experience.
• Robot VisionWhen robots operate in complex environments, they may encounter situations where objects are partially occluded. Amodal3R can help robots perceive and understand objects in their environment more accurately, enabling them to better plan paths and execute tasks.
• autonomous drivingIn the field of autonomous driving, vehicles need to perceive objects in their surroundings in real time. Amodal3R can reconstruct complete 3D models from partially occluded images, helping autonomous driving systems to more accurately identify and handle complex traffic scenarios.
• 3D asset creationIn game development, film production, and other fields that require 3D assets, Amodal3R can generate high-quality 3D models from simple 2D images, simplifying the 3D modeling process.
• academic researchAmodal3R provides new tools and methods for research in computer vision and 3D reconstruction. Researchers can use the model to explore more complex scenes and more efficient reconstruction algorithms.