Stable Video 3D (SV3D) - a multi-view compositing and 3D generation model, developed by Stability AI.

What is Stable Video 3D?

Stable Video 3D (SV3D) is a multi-view synthesis and 3D generation model developed by Stability AI. It can generate consistent multi-view images from a single image and further optimize the generation of high-quality 3D mesh models. This model improves upon the previously released Stable Video Diffusion model, providing better quality and a more immersive multi-view experience. Compared to other 3D generation models, SV3D's main advantage lies in its use of a video diffusion model instead of an image diffusion model, which offers significant benefits in terms of generalization and viewpoint consistency of the generated output.

Features of Stable Video 3D

• Multi-view video generationSV3D can generate video content from multiple perspectives from a single image input. Users can view objects from different directions and angles, each perspective is of high quality, and consistency is maintained between perspectives.
• 3D Mesh CreationBy using generated multi-view videos and Stable Video 3D models, users can create 3D meshes of objects. These 3D meshes are inferred from 2D images and can be used in various 3D applications such as game development, virtual reality, and augmented reality.
• Track video generationStable Video 3D provides the ability to generate orbital videos around objects, allowing users to create videos that rotate or move around objects, providing a dynamic perspective experience.
• Camera path controlSV3D supports creating 3D videos along a specified camera path, allowing users to precisely control the video's perspective and camera movement, providing greater freedom for creation.
• New Perspective Synthesis (NVS)SV3D has made significant progress in new perspective compositing, enabling the generation of consistent and realistic views from any given angle, thus improving the realism and accuracy of 3D generation.

Stable Video 3D official website entrance

• Official project homepage:https://sv3d.github.io/
• Technical Report:https://stability.ai/s/SV3D_report.pdf
• Hugging Face Model:https://huggingface.co/stabilityai/sv3d

How Stable Video 3D Works

1. New Perspective Synthesis (NVS):
   • Input imageThe user provides a single 2D image as input, which contains one or more objects.
   • Camera attitude controlDefine a camera trajectory, including a series of angles (elevation and azimuth), to control the viewpoint of the generated image.
   • Potential video diffusion modelThe method uses a trained latent video diffusion model (such as Stable Video Diffusion – SVD) that generates a series of new viewpoint images based on the input image and camera pose. These images simulate orbital video around a 3D object.
2. 3D representation optimization:
   • Rough 3D ReconstructionUsing the generated multi-view images as targets, a coarse representation of the 3D object is reconstructed by training a NeRF (Neural Radiance Fields) model. This step is performed at a lower resolution to capture the object's approximate shape and texture.
   • Mesh ExtractionExtracting a preliminary 3D mesh from a trained NeRF model, typically using the Marching Cubes algorithm.
   • Fine optimizationThe 3D mesh is further refined using DMTet (Deep Marching Tetrahedra) representation. This step is performed at high resolution to improve the accuracy of details and the quality of the mesh.
3. Improved 3D optimization technology:
   • Masked fractional distillation sampling (SDS) lossTo improve the 3D quality of invisible regions, SV3D introduces a masking fractional distillation sampling loss. This loss function focuses on filling in and optimizing regions that are not visible in the reference viewpoint during training.
   • Decoupled lighting modelSV3D also proposes a decoupled lighting model that is optimized independently of 3D shape and texture to reduce rendering problems caused by fixed lighting conditions.
4. Training and assessment:
   • DatasetSV3D is trained on datasets containing diverse 3D objects, such as the Objaverse dataset.
   • EvaluateThe quality of multi-view images and 3D meshes generated by SV3D is evaluated by comparing them with real-world 3D data and other NVS methods.