MVGenMaster - A multi-view diffusion model jointly developed by Fudan University, Alibaba, and other laboratories.

What is MVGenMaster?

MVGenMaster is a multi-view diffusion model jointly developed by Fudan University, Alibaba DAMO Academy, and Hupan Lab. It leverages enhanced 3D priors to handle diverse novel view synthesis (NVS) tasks. The model utilizes 3D priors that measure depth and camera pose distortion to improve NVS generalization and 3D consistency. MVGenMaster can generate up to 100 new views from a single image using a single forward propagation. The research team has released a large-scale multi-view image dataset, MvD-1M, containing 1.6 million scenes, and employed multiple training and model optimization techniques to enhance the model's performance on large-scale datasets.

Main functions of MVGenMaster

• Multi-view generationGenerate up to 100 new views from a single image or multiple reference images, suitable for different NVS tasks, such as single-view NVS, two-view interpolation, and flexible NVS with arbitrary reference and target views.
• 3D Prior IntegrationUsing 3D priors that measure depth and camera pose distortion, the model maintains a consistent 3D structure in the 2D diffusion model.
• Flexibility and generalizationThe model design is flexible and can adapt to different perspectives and scenarios, demonstrating generalization ability in a variety of scenarios.
• High-efficiency forward processIt generates multiple views in a single forward pass without the need for complex iterative reasoning or dataset updates.
• Large-scale dataset supportCombines the MvD-1M dataset, which contains 1.6 million scenes and well-aligned metric depth.

MVGenMaster's technical principles

• 3D PriorMVGenMaster uses depth measurement and camera pose to create 3D priors, which are then used in the model to guide the generation of new views and ensure the consistency of the 3D structure.
• Geometric TwistBased on the geometric warp function, the model warps the pixel and canonical coordinate mapping (CCM) of the reference view from the source view to the target view.
• Multi-view diffusion model (LDM)MVGenMaster is based on the Latent Diffusion Model (LDM) and learns how to synthesize images of a target view from reference images and 3D priors.
• Attention mechanismThe model uses a full attention mechanism to capture dense camera pose representations across all reference and target views.
• Plücker raysPlücker rays are used to represent camera pose, providing the model with accurate camera position and orientation information.
• Key rescaling technologyTo handle extremely long sequences of target views, MVGenMaster introduces a key rescaling technique to enhance guidance from the reference view and balance the attention dilution problem.

MVGenMaster's project address

• Project official website:ewrfcas.github.io/MVGenMaster
• GitHub repository:https://github.com/ewrfcas/MVGenMaster
• arXiv technical paper:https://arxiv.org/pdf/2411.16157

Application scenarios of MVGenMaster

• video gamesIn video games, it is used to generate high-quality 3D content, enhancing the realism and immersion of the game visuals.
• Movies and visual effectsIn film production and visual effects, it creates complex 3D scenes and effects, reducing the cost of actual shooting and post-production.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it generates realistic 3D environments, providing users with a richer and more interactive experience.
• 3D modeling and designDesigners can create 3D models from 2D images, accelerating the product design and prototyping process.
• Architectural VisualizationIn architecture and urban planning, it helps architects and planners present architectural designs from different perspectives and conduct better evaluations of different options.