GEN3C - A generative video model developed by NVIDIA in collaboration with the University of Toronto and others.
GEN3C is a new generative video model developed by NVIDIA, the University of Toronto, and the Vector Institute. It generates high-quality 3D video content based on precise camera control and spatiotemporal consistency. GEN3C builds a point cloud-based 3D cache...
What is GEN3C?
GEN3C is a novel generative video model developed by NVIDIA, the University of Toronto, and the Vector Institute. It generates high-quality 3D video content based on precise camera control and spatiotemporal consistency. GEN3C constructs a point cloud-based 3D cache to guide video generation, generates 3D scenes by backprojecting depth estimates from input images or video frames, and renders 2D video based on user-provided camera trajectories as conditional input to the generative model. GEN3C's core advantage lies in its precise control of camera motion, avoiding the inconsistencies caused by the lack of explicit 3D modeling in traditional methods. GEN3C supports video generation from single-view to multi-view perspectives, applicable to both static and dynamic scenes, and achieves state-of-the-art results in novel view compositing tasks with sparse views. GEN3C supports 3D editing and the generation of complex camera movements (such as push-in and pull-out shots), providing powerful tools for video creation and simulation.
Main functions of GEN3C
- Precise camera controlIt generates videos based on user-specified camera trajectories, supports complex camera movements (such as zooming in and out, rotating the camera, etc.), and maintains the spatiotemporal consistency of the video.
- 3D Consistent Video GenerationIt supports the generation of realistic and consistent videos, avoiding problems such as objects suddenly appearing or disappearing.
- Novel view synthesis with multiple and sparse perspectivesIt supports inputs from single-viewpoint, sparse multi-viewpoint to dense multi-viewpoint, generating high-quality new types of view videos.
- 3D editing and scene manipulationUsers can modify 3D point clouds (such as deleting or adding objects) to edit the scene and generate corresponding videos.
- Long video generationSupports the generation of long videos while maintaining spatiotemporal consistency.
GEN3C's technical principles
- Building a 3D cacheThe depth estimation of the input image or video frame is back-projected to generate a 3D point cloud, forming a spatiotemporally consistent 3D cache. The cache serves as the basis for video generation, providing an explicit 3D structure of the scene.
- Rendering 3D cacheBased on the camera trajectory provided by the user, render the 3D cache into a 2D video.
- Video generationThis approach uses a pre-trained video diffusion model (such as Stable Video Diffusion or Cosmos) as conditional input to generate high-quality videos. The model optimizes the denoising objective during the diffusion process, repairing rendering artifacts and filling in missing information.
- Multi-perspective fusionWhen the input contains multiple perspectives, GEN3C uses a max-pooling fusion strategy to aggregate information from different perspectives into the video generation model, generating a consistent video.
- Autoregressive generation and cache updateFor long video generation, GEN3C divides the video into multiple overlapping blocks, generates the video block by block, and updates the 3D cache to maintain the spatiotemporal consistency of the video.
GEN3C's project address
- Project official website:https://research.nvidia.com/labs/toronto-ai/GEN3C/
- GitHub repository:https://github.com/nv-tlabs/GEN3C
- arXiv technical paper:https://arxiv.org/pdf/2503.03751
Application scenarios of GEN3C
- Single-view video generationGenerate dynamic videos from a single image, ideal for rapid content creation.
- New type of view synthesisGenerate new perspective videos from a limited number of viewpoints for use in VR/AR and 3D reconstruction.
- Driving SimulatorGenerates videos of driving scenarios from different perspectives to aid in autonomous driving training.
- Dynamic video re-renderingGenerate new perspectives for existing videos, used in video editing and secondary creation.
- 3D scene editing: Generate new videos after modifying scene content, assisting in film and television production and game development.