Depth Anything 3 - A Visual Spatial Reconstruction Model Launched by ByteDance

What is Depth Anything 3?

Depth Anything 3 (DA3) is a visual space reconstruction model developed by ByteDance's Seed team. It utilizes a single Transformer architecture to recover 3D spatial geometry from visual input from any viewpoint. The model employs a "depth-ray" representation method, eliminating the need for complex multi-task training and simplifying model design. Depth Anything 3 surpasses previous mainstream models in camera pose accuracy and geometric reconstruction accuracy while maintaining high inference speed. The model is suitable for applications such as autonomous driving, robot navigation, and virtual reality, providing a new and efficient solution for visual space reconstruction.

Main features of Depth Anything 3

• Multi-view spatial reconstructionDepth Anything 3 (DA3) can reconstruct three-dimensional spatial structures from any number of visual inputs, such as a single image, multi-view images, or video streams.
• Camera pose estimationThe model can accurately estimate the camera pose (including position and orientation) of the input image, even without knowing the camera parameters.
• Monocular Depth EstimationThe model performs well on monocular depth estimation tasks, predicting pixel-level depth information from a single image, providing fundamental support for 3D scene understanding.
• New Perspective SynthesisBy combining with 3D Gaussian rendering technology, the model can generate high-quality images viewed from unknown perspectives, suitable for perspective rendering tasks in virtual reality and augmented reality.
• Efficient Reasoning and DeploymentThe model's simple architecture design gives it significant advantages in inference speed and resource consumption, enabling it to quickly handle large-scale scenarios and is suitable for deployment on mobile and embedded devices.

Technical Principles of Depth Anything 3

• Single Transformer architectureIt uses a single Transformer model (such as DINOv2) as its basic architecture, eliminating the need for complex custom designs. The Transformer's self-attention mechanism can flexibly handle any number of input views, dynamically exchange cross-view information, and achieve efficient global space modeling.
• Depth-ray characterizationThe model proposes a "depth-ray" representation method that comprehensively describes 3D space by predicting depth maps and ray maps. The depth map provides the distance from pixels to the camera, while the ray map describes the projection direction of pixels in 3D space. This representation method naturally decouples spatial geometry from camera motion, simplifies model output, and improves both accuracy and efficiency.
• Input-adaptive cross-view self-attention mechanismThis introduces an input-adaptive cross-view self-attention mechanism, which achieves efficient cross-view information exchange by dynamically rearranging the tokens of the input views. This mechanism enables the model to flexibly handle various input scenarios from monocular to multi-view.
• Dual DPT head designTo jointly predict depth and ray maps, DA3 employs a dual DPT head structure. The two prediction heads share a feature processing module, optimizing the depth and ray map outputs separately during the final fusion stage, enhancing the interaction and consistency between the two tasks.
• Teacher-student training paradigmThe teacher-student training paradigm is adopted, which generates high-quality pseudo-labels by a teacher model trained on synthetic data, providing more accurate supervision for the student model.
•  Achieve high-precision output in one stepThis method generates high-precision depth and ray maps with a single feedforward, eliminating the need for multiple iterative optimizations required in traditional methods. This design significantly improves inference speed, simplifies training and deployment processes, and ensures both accuracy and efficiency in 3D reconstruction.

Project address for Depth Anything 3

• Project official websitehttps://depth-anything-3.github.io/
• GitHub repositoryhttps://github.com/ByteDance-Seed/depth-anything-3
• arXiv technical paperhttps://arxiv.org/pdf/2511.10647
• Experience the demo online:https://huggingface.co/spaces/depth-anything/depth-anything-3

Application scenarios of Depth Anything 3

• autonomous drivingThe DA3 can quickly reconstruct the 3D environment from multi-view images captured by vehicle cameras, helping autonomous driving systems to more accurately perceive the distance and position of surrounding objects, thereby improving the reliability and safety of decision-making.
• Robot NavigationBy reconstructing the three-dimensional structure of the environment in real time, DA3 can provide robots with accurate terrain and obstacle information, supporting them to perform efficient navigation and path planning in complex environments.
• Virtual Reality (VR) and Augmented Reality (AR)It can quickly convert real-world scenes into high-precision 3D models for scene reconstruction in virtual reality or fusion of virtual objects in augmented reality, thereby enhancing the user's sense of immersion.
• Architectural surveying and designIt reconstructs detailed 3D point clouds from multi-view images of architectural scenes, providing efficient data support for architectural surveying, interior design, and virtual architectural roaming.
• Cultural heritage protectionDA3 can be used to reconstruct the three-dimensional structure of historical buildings or cultural relics, facilitating digital protection, restoration research, and virtual display, and helping to preserve and promote cultural heritage.