AuraFusion360 - a 3D scene restoration technology that enables high-quality object removal and hole filling.
AuraFusion360 is a novel reference-based method for 360° borderless scene restoration, primarily used in fields such as virtual reality and architectural visualization. It achieves high-quality restoration of 3D scenes represented by Gaussian scattering...
What is AuraFusion360?
AuraFusion360 is a novel reference-based method for 360° borderless scene restoration, primarily used in fields such as virtual reality and architectural visualization. It achieves high-quality object removal and hole filling through Gaussian scattering representation of 3D scenes. Core features include: depth-aware unseen region mask generation for accurate identification of occluded areas; Adaptive Guided Depth Diffusion (AGDD), a zero-shot method requiring no additional training for accurate initial point placement; and SDEdit-based detail enhancement to ensure consistency across multiple viewpoints.
Main functions of AuraFusion360
- High-quality repairWhile maintaining geometric accuracy, it significantly outperforms existing methods, adapts to dramatic changes in perspective, and demonstrates superior perceptual quality.
- High-quality multi-view consistencyAuraFusion360 ensures multi-view consistency under significant viewpoint changes by combining explicit representations of Gaussian point clouds with the generative capabilities of diffusion models.
- Object Removal and Gaussian Point Cloud OptimizationAuraFusion360 can directly remove objects with occlusion properties from Gaussian point clouds, achieving high-quality scene restoration by optimizing unprojected Gaussian point clouds.
- Supports the repair of complex geometriesSuitable for scene restoration with complex geometries, effectively handling large viewpoint changes and significant occlusion. It utilizes depth-aware mask generation for unseen areas and adaptively guided depth diffusion.
AuraFusion360's technical principles
- Depth perception unseen region mask generationThis method uses depth map deformation technology to identify occluded areas in the scene and generate masks for these unseen areas. Based on monocular depth estimation and geometric alignment, it ensures the accuracy of the repaired areas.
- Adaptive Guided Depth Diffusion (AGDD)AGDD is a zero-shot method that aligns estimated monocular depth with existing geometry without additional training. By adaptively adjusting depth information, it ensures geometric consistency of the repaired scene.
- Enhanced details based on SDEditEnhance details and ensure consistency across multiple viewpoints using SDEdit-guided enhancement. SDEdit, based on RGB guidance, preserves reference view information, retaining scene details and textures during the repair process.
- 360-USID dataset supportAuraFusion360 introduces the 360-USID dataset, the first comprehensive dataset for 360° borderless scene restoration, which includes real ground data.
AuraFusion360 project address
- Project official website:https://kkennethwu.github.io/aurafusion360/
- Github repository:https://github.com/kkennethwu/AuraFusion360_official
- arXiv technical paper:https://arxiv.org/pdf/2502.05176
Application scenarios of AuraFusion360
- Virtual scene editingIn VR environments, it's necessary to remove or replace certain objects in the scene. AuraFusion360 can effectively repair the holes left behind by these removed objects, ensuring the integrity and consistency of the scene.
- Building model optimizationIn architectural models, it's necessary to remove unwanted objects, such as temporary support structures or construction materials. AuraFusion360 can help optimize these models, generating clearer and more accurate architectural visualizations.
- Special effects productionIn film and television special effects, it is necessary to remove or replace certain objects in a scene, such as slings, scaffolds, or temporary props. AuraFusion360 can restore these areas with high quality, generating more realistic special effects.
- Cultural Relics RestorationIn cultural heritage preservation, it is necessary to repair or rebuild damaged artifacts or historical buildings. AuraFusion360 can help generate high-quality restoration results, ensuring the integrity and historical value of the artifacts.