Flex3D - A two-stage 3D generation framework jointly developed by Meta GenAI and Oxford University

What is Flex3D?

Flex3D is an innovative two-stage 3D generation framework developed by the GenAI team at Meta and a research team at Oxford University. It addresses the challenge of generating high-quality 3D content from text, single images, or sparse view images based on an arbitrary number of high-quality input views. In the first stage, a diverse range of candidate views are generated based on a fine-tuned multi-view and video diffusion model. A view selection mechanism ensures that only high-quality and consistent views are used for reconstruction. In the second stage, the Flexible Reconstruction Model (FlexRM), a transformer-based model, handles an arbitrary number of input views and directly outputs 3D Gaussian points for efficient and detailed 3D generation. Flex3D achieves state-of-the-art performance in 3D reconstruction and generation tasks, with a win rate exceeding 92% in user studies.

Main functions of Flex3D

• High-quality 3D content generationGenerate high-quality 3D models from text descriptions, single images, or sparse view images.
• Flexible view generationBased on a fine-tuned multi-view and video diffusion model, it generates rich candidate views and captures multiple angles of the target 3D object.
• View filtering mechanismAutomatically selects high-quality and consistent views for use in subsequent 3D reconstruction processes.
• Flexible Reconstruction Model (FlexRM)Based on the transformer architecture, it can process any number of input views and directly output 3D Gaussian points.
• Highly efficient 3D representationUsing three-plane representation and 3D Gaussian rendering techniques, we can achieve fast and detailed 3D model generation.
• robustnessTraining by simulating imperfect input views enhances the model's robustness to input noise.

The technical principles of Flex3D

• Multi-view diffusion model: Generate a candidate view pool using a fine-tuned multi-view image diffusion model and a video diffusion model.
• View Selection PipeBased on quality assessment and feature matching networks, high-quality views are selected for use in 3D reconstruction.
• Converter architectureFlexRM is based on a transformer architecture and can handle different numbers and angles of input views.
• Three-plane representation and 3D Gaussian plottingCombining three-plane features and 3D Gaussian rendering technology, the three-plane features are decoded into 3D Gaussian points based on MLP.
• Two-stage training strategyFirst, the model is pre-trained, and then a second stage of training is performed using densely rendered real-world data.
• Imperfect Input View SimulationDuring training, imperfect input views are simulated, and noise is added to 3D Gaussian points to enhance the robustness of the model.

Flex3D project address

• Project official website:junlinhan.github.io/projects/flex3d
• arXiv technical paper:https://arxiv.org/pdf/2410.00890

Application scenarios of Flex3D

• Video game developmentIt can quickly generate 3D models of game assets and environments, improving the efficiency of game design and development.
• Augmented Reality (AR) and Virtual Reality (VR)Create realistic 3D objects and scenes for AR and VR applications to enhance user immersion.
• Film and animation productionGenerate high-quality 3D models for use in film and animation, including character design, prop making, and scene construction.
• RoboticsGenerate 3D models to simulate and train robots' navigation and maneuvering capabilities in complex environments.
• e-commerceGenerate 3D views of products for online stores, providing richer product displays and helping consumers make purchasing decisions.