AB
AB
AiBoss

All Categories

project

ReCamMaster - A video re-rendering framework jointly developed by Zhejiang University and Kuaishou, among others.

ReCamMaster is a video re-rendering framework jointly developed by Zhejiang University, Kuaishou Technology, and others. It can regenerate video content based on new camera trajectories. Through pre-trained models and frame-dimensional conditional mechanisms, combined with multi-camera synchronization data...

What is ReCamMaster?

ReCamMaster is a video re-rendering framework jointly developed by Zhejiang University, Kuaishou Technology, and others. It can regenerate video content based on new camera trajectories. Through pre-trained models and frame-dimensional conditional mechanisms, combined with multi-camera synchronization datasets and camera pose conditions, it enables flexible adjustment of video perspective and motion trajectory. Users can upload videos and specify trajectories, and the system will generate videos with new perspectives. It is widely used in video creation, post-production, education, and other fields, bringing new perspectives and dynamic effects to video content, and improving creative freedom and quality.

ReCamMaster's main functions

  • Camera trajectory controlled video re-renderingIt can re-render the input video based on the camera trajectory specified by the user, generating dynamic scenes with new perspectives while maintaining the appearance consistency with the original video.
  • Video stabilizationConverts unstable video into smooth video while preserving the original scene and motion.
  • Video super-resolution and expansionThe framework can achieve local super-resolution of video by inputting a zoom trajectory, generating clearer details. It can also generate content beyond the original video's field of view by inputting a zoom-out trajectory.
  • Supports complex trajectory inputIt can handle various complex trajectories such as translation, rotation, and scaling, and users can customize camera motion.
  • High-quality video generationBy using frame-level conditional stitching technology, we maintain the dynamic synchronization and multi-frame consistency of the video, generating high-quality video.

ReCamMaster's Technical Principles

  • Pre-trained text-to-video diffusion modelReCamMaster uses a pre-trained text-to-video diffusion model as its infrastructure. The model consists of a 3D variational autoencoder (VAE) and a Transformer-based diffusion model (DiT) that can generate high-quality video content.
  • Frame Dimension Conditional MechanismThe framework proposes an innovative frame-dimensional conditional mechanism that concatenates the tags of the source and target videos along the frame dimension, using them as input to the diffusion Transformer. This enables the model to better understand the spatiotemporal relationships between video pairs, generating a target video that is synchronized and consistent with the source video.
  • Camera attitude conditionsReCamMaster encodes the target camera trajectory into video features using a learnable camera encoder. The camera trajectory is represented as a rotation and translation matrix and projected onto the same channel as the video markers, enabling flexible control over the camera trajectory.
  • Multi-camera synchronized video datasetTo overcome the problem of scarce training data, the research team used Unreal Engine 5 to build a large-scale multi-camera synchronized video dataset. The dataset contains diverse scenes and camera motions, which helps the model generalize to real-world video.
  • Training strategyReCamMaster employs strategies such as fine-tuning key components, applying noise, and unifying camera control tasks during training. These strategies help improve the model's generalization and generative capabilities while reducing the domain gap between synthetic and real data.

ReCamMaster's project address

Application scenarios of ReCamMaster

  • Video creation and post-productionReCamMaster can re-render videos based on user-specified camera paths, generating video content with new perspectives and motion trajectories. Creators can adjust the camera path to add more creative camera movements and enhance the visual effects of their videos.
  • Video stabilizationReCamMaster can convert shaky handheld video into smooth, stable video while preserving the original scene and motion.
  • Autonomous driving and robot visionReCamMaster can be used to generate driving scenarios from different perspectives, helping to train autonomous driving models and improve their adaptability to complex scenarios.
  • Virtual Reality and Augmented RealityReCamMaster can generate video content that matches the virtual environment, providing richer visual materials for virtual reality and augmented reality applications.
Back