ReCapture - A video processing technology jointly developed by Google and the National University of Singapore

What is ReCapture?

ReCapture, a video processing technology developed by Google and the National University of Singapore, can generate new videos with novel camera trajectories from a single user-provided video. ReCapture uses a multi-view diffusion model or depth-based point cloud rendering to generate noisy anchor videos with new camera trajectories. It then employs masked video fine-tuning to transform the anchor videos into clean, time-consistent re-angled videos, preserving scene motion from the original video and showcasing the scene from a new angle. ReCapture can reasonably imagine scenes that were not visible in the original video.

ReCapture's main functions

• Generate new perspective videosGenerates videos with entirely new camera trajectories from a user-provided source video, supporting observation of the same scene from different angles.
• Preserve the original scene movementWhile generating new perspective videos, it preserves all existing scene motion in the source video.
• Cinematic camera movementSimulates cinematic camera movements, such as zoom, pan, and tilt, to enhance the visual effects of videos.
• Scene completionImagine and fill in the missing scenes in the source video to enhance the completeness of the video content.
• Improve video qualityBased on masked video fine-tuning technology, noisy anchor videos are converted into clean, time-consistent, high-quality videos.

The technical principle of ReCapture

• Anchor video generation:
  • Depth estimation and point cloud renderingBased on frame-by-frame depth estimation, video frames are converted into 3D point cloud sequences. New perspectives are simulated according to user-specified camera motion, and the point cloud sequences are rendered to generate new video frames.
  • Multi-view diffusion modelFor more complex camera trajectories (such as trajectories around a point in the scene), a multi-view diffusion model is used to generate video frames from new perspectives.
• Masked video fine-tuning:
  • Temporal LoRA (Low-Rank Adaptation)Fine-tune the temporal LoRA learning scene dynamics on the masked anchor video, focusing on the meaningful pixel portions of the learning anchor video and ignoring unknown areas.
  • LoRA SpaceFine-tune the appearance of the spatial LoRA learning scene on the enhanced frames of the source video to ensure that the filled pixels blend seamlessly with the original video pixels.
• Strong priors for video modelsUsing strong prior knowledge from the video model, appropriate content is automatically filled into the mask area, significantly improving the temporal consistency of the video and eliminating jitter in the anchor video.

ReCapture's project address

• Project official website:generative-video-camera-controls.github.io
• arXiv technical paper:https://arxiv.org/pdf/2411.05003

Application scenarios of ReCapture

• Film and video productionFilmmakers re-edit and adjust already shot video, changing the original camera angles and movements to create new visual effects or improve scene composition.
• Video editing and post-productionVideo editors can modify or enhance video content, for example, by changing the camera angle to highlight key elements in the video or removing unwanted background.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it generates more immersive and interactive video content, providing the ability to observe scenes from different perspectives.
• News and documentariesJournalists and documentary filmmakers recreate events, showcasing news stories or historical events from multiple perspectives, thus increasing the depth and dimension of their reporting.
• Live sports broadcastsLive broadcasts of sports events offer more perspectives, allowing viewers to experience the game from different camera angles and enhancing the viewing experience.