AB
AB
AiBoss

All Categories

project

SynCamMaster - A multi-view video generation model developed by Kuaishou in collaboration with institutions such as Zhejiang University and Tsinghua University.

SynCamMaster is the world's first multi-view video generation model, developed through a collaboration between researchers from Zhejiang University, Kuaishou Technology, Tsinghua University, and the Chinese University of Hong Kong. It can combine 6-DOF camera poses to generate open-ended videos from any viewpoint...

What is SynCamMaster?

SynCamMaster, a collaborative project between researchers from Zhejiang University, Kuaishou Technology, Tsinghua University, and the Chinese University of Hong Kong, is the world's first multi-view video generation model. It combines 6-DOF camera poses to generate open-world videos from any viewpoint. SynCamMaster enhances pre-trained text-to-video models to ensure content consistency across different viewpoints and supports multi-camera video generation. Based on plug-in modules and a multi-view synchronization module, it achieves dynamic synchronization between viewpoints, maintaining 4D consistency. SynCamMaster can be extended to new-view video synthesis, re-rendering the input video for viewing from a new perspective.

SynCamMaster's main functions

  • Multi-view video generationSynCamMaster can generate multiple videos from different perspectives of the same dynamic scene.
  • Dynamic synchronization between viewpointsMaintain dynamic synchronization across multiple perspectives to ensure consistency in time and space between video content generated by different cameras.
  • Open World Video GenerationSynCamMaster supports generating open-world videos from any perspective.
  • 6-DOF camera poseCombined with 6 degrees of freedom (6 DoF) camera pose, users can capture scenes from any angle.
  • Pre-trained model enhancementEnhance pre-trained text-to-video models with plug-and-play modules for multi-camera video generation.
  • New Perspective Video SynthesisSynCamMaster can be extended to new perspective video synthesis, introducing reference videos into multi-camera video generation models to re-render input videos from new perspectives.

SynCamMaster's technical principles

  • Pre-trained text-to-video modelBased on a pre-trained text-to-video diffusion model, it generates consistent 3D video content according to text descriptions.
  • Multi-view synchronization moduleA multi-view synchronization module is introduced, which is integrated into each Transformer block to maintain the appearance and geometric consistency between different viewpoints.
  • Camera encoderThe camera encoder encodes the camera's extrinsic parameters (such as rotation matrices and translation vectors) into an embedding space of the same dimension as the spatial features.
  • Cross-perspective self-attentionIn the multi-view synchronization module, features from different perspectives are aggregated using a cross-view self-attention layer to achieve information exchange and synchronization between views.
  • Mixed training datasetsDesign a hybrid training scheme that combines multi-camera images, monocular video, and multi-camera video rendered by Unreal Engine.
  • Progressive training strategyIntroducing a progressive training strategy, gradually increasing the relative angle difference between different viewpoints during training, thereby improving the model's performance when dealing with large viewpoint differences.

SynCamMaste's project address

Application scenarios of SynCamMaster

  • Virtual shootingIn film and video production, it generates videos from multiple angles, helping directors and production teams preview scenes and achieve complex shot combinations.
  • Game developmentGame developers create dynamic game trailers or demo videos to showcase the game environment and character interactions from different perspectives.
  • Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, generate dynamic content that changes in the user's perspective to enhance the immersive experience.
  • Simulation trainingFor scenarios requiring multi-angle observation, such as driving simulation and surgical simulation, it provides real-time videos from different perspectives to enhance training effectiveness.
  • Monitoring systemIn the field of security monitoring, integrating video streams from multiple cameras to generate video from a unified perspective makes it easier for monitoring personnel to better understand the development of events.
Back