AB
AB
AiBoss

All Categories

project

Matrix3D - A unified photogrammetry model jointly developed by Nanjing University, Apple, and Hong Kong University of Science and Technology.

Matrix3D is a novel unified photogrammetry model developed through a collaboration between Nanjing University, Apple Inc., and Hong Kong University of Science and Technology. It can perform multiple photogrammetry sub-tasks within a single model, including pose estimation, depth prediction, and new view merging...

What is Matrix3D?

Matrix3D is a novel unified photogrammetry model developed through a collaboration between Nanjing University, Apple Inc., and Hong Kong University of Science and Technology. It can perform multiple photogrammetric subtasks within a single model, including pose estimation, depth prediction, and new view synthesis. At its core is the Multimodal Diffusion Transformer (DiT), which integrates data from multiple modalities, including images, camera parameters, and depth maps, to achieve flexible task processing. Matrix3D employs a mask learning strategy during training, enabling full-modal training based on bimodal data (such as image-pose or image-depth pairs) even when some data is missing, significantly increasing the amount of available training data. Matrix3D supports multi-turn interaction, allowing users to progressively input information to optimize the generated results, providing strong flexibility in 3D content creation.

Main functions of Matrix3D

  • Pose estimationIt can estimate the camera pose from images with sparse viewpoints and accurately predict the relative position and orientation of the camera when the image overlap is low.
  • Deep PredictionIt can predict high-quality depth maps from monocular or multi-view images. The model supports generating depth information from a small number of images, which can be used for subsequent 3D reconstruction tasks.
  • Novel View SynthesisIt can generate new view images in any pose based on input images. It supports synthesizing new viewpoints from single or multiple images.
  • 3D ReconstructionCombining the results of pose estimation, depth prediction, and new view synthesis, Matrix3D can be used for 3D reconstruction of single or small numbers of images. By integrating with 3D Gaussian sputtering (3DGS) optimization, the model can generate high-quality 3D point clouds.
  • Multi-task interaction and flexibilityMatrix3D supports flexible input/output configuration and can dynamically adjust tasks based on information provided by the user.
  • Mask learning and data utilizationBy employing a mask learning strategy, Matrix3D can be trained on partially complete data, significantly increasing the range of available training data and improving the model's generalization ability.

The technical principles of Matrix3D

  • Multimodal diffusion converterMatrix3D uses a diffusion transformer architecture that can process multimodal data (such as RGB images, camera poses, and depth maps) and unify them into a two-dimensional representation, enabling cross-modal feature fusion and generation.
  • Mask learning strategyThe model employs mask learning during training, randomly masking the input data to learn how to recover a clean output from noisy data. This allows it to handle incomplete input data, significantly increasing the amount of usable training data.
  • Unified probability modelMatrix3D is based on a unified probabilistic model and can dynamically adjust task requirements through flexible input/output configuration, supporting a variety of photogrammetric tasks such as pose estimation, depth prediction, and new view synthesis.
  • 3D Gaussian Sputtering OptimizationThe generated depth map and new view image can be used to initialize 3D Gaussian sputtering (3DGS) optimization to achieve high-quality 3D point cloud reconstruction.
  • Data normalizationThrough scene normalization and camera normalization techniques, Matrix3D can handle the diverse distribution of different datasets, ensuring that data of different modalities are processed under a unified scale and coordinate system.

Matrix3D project address

Application scenarios of Matrix3D

  • Virtual Reality (VR) and Augmented Reality (AR)Matrix3D can generate high-quality new views and depth information from a small number of 2D images, providing rich 3D scenes and object models for VR and AR applications.
  • Game developmentMatrix3D can quickly generate geometric information for 3D scenes and objects, helping game developers quickly create 3D models for games from concept art or a small number of reference images.
  • Film and television productionMatrix3D can be used to generate complete 3D scenes from a small number of 2D images, facilitating post-production special effects and virtual photography.
  • Architectural DesignMatrix3D can generate 3D architectural models from architectural drawings or a small number of photos, helping designers quickly validate designs and present them to clients.
  • Virtual try-onMatrix3D can generate 3D models of a user's body for virtual try-on of clothing and accessories.
Back