TrajectoryCrafter - A monocular video free-movement camera technology launched by Tencent and CUHK
TrajectoryCrafter, developed by Tencent PCG ARC Lab and the Chinese University of Hong Kong, is a camera trajectory redirection method for monocular video. It allows for free adjustment of the camera position and angle in post-production, easily altering the video's trajectory...
What is TrajectoryCrafter?
TrajectoryCrafter, developed by Tencent PCG ARC Lab and the Chinese University of Hong Kong, is a camera trajectory redirection method for monocular video. It allows for flexible adjustment of camera position and angle in post-production, easily altering camera movements within the video. Based on decoupled view transformation and content generation, TrajectoryCrafter uses a two-stream conditional video diffusion model, taking point cloud rendering and source video as conditions to achieve precise control over user-specified camera trajectories and high-quality 4D content generation. TrajectoryCrafter employs an innovative dual reprojection strategy and a hybrid dataset (combining dynamic monocular video and static multi-view data) to train its model, significantly improving its generalization ability across diverse scenarios. TrajectoryCrafter performs exceptionally well on multi-view and large-scale monocular video datasets, generating high-fidelity, source-consistent trajectory videos, providing new possibilities for immersive video experiences.
The main functions of TrajectoryCrafter
- Precise trajectory controlThe user specifies any camera trajectory (such as translation, rotation, scaling, etc.) and generates video content that matches it.
- High-fidelity video generationThe generated video is visually consistent with the original video, with high-quality details and textures.
- 4D ConsistencyThe generated video is spatially consistent with the target trajectory and temporally coherent with the original video, avoiding content drift or flickering.
- Diverse scenario generalizationThe model can adapt to various scenarios, including indoor, outdoor, and dynamic scenes, and has good generalization ability.
The technical principles of TrajectoryCrafter
- Two-stream conditional video diffusion model:
- Decoupling view transformation and content generationThis approach separates the deterministic transformation of camera trajectory from the randomness of content generation. It achieves precise view transformations based on point cloud rendering and generates high-quality content using a video diffusion model.
- Dual-stream conditional mechanismThe model contains two conditional inputs: point cloud rendering (used for precise control of view transformations) and source video (used to provide details and textures). A unique Ref-DiT module (Reference Conditional Diffusion Transformer) is used to inject detailed information from the source video into the generation process through a cross-attention mechanism, improving the fidelity of the generated video.
- Dynamic point cloud renderingDepth estimation converts monocular video into dynamic point clouds, rendering a new view based on user-specified camera trajectories. Point cloud rendering accurately captures geometric relationships and view transformations, providing geometric guidance.
- Mixed datasets and training strategiesA hybrid dataset strategy is employed, combining network-scale monocular video and static multi-view datasets for training. For monocular video, a dual reprojection strategy is used to generate large-scale training samples. Specifically, the video is upscaled to a point cloud through depth estimation, the new view is rendered, and then reprojected back to the original view, simulating the effect of point cloud rendering. The model uses a two-stage training strategy: the first stage focuses on the accuracy of view transformations and the synthesis of missing regions; the second stage uses a multi-view dataset to improve the consistency between the generated video and the source video.
TrajectoryCrafter's project address
- Project official website:https://trajectorycrafter.github.io/
- GitHub repository:https://github.com/TrajectoryCrafter/TrajectoryCrafter
- arXiv technical paper:https://arxiv.org/pdf/2503.05638
- Experience the demo online:https://huggingface.co/spaces/Doubiiu/TrajectoryCrafter
Application scenarios of TrajectoryCrafter
- Immersive EntertainmentUsed in VR/AR, it allows users to freely switch perspectives, enhancing immersion.
- Creative Video ProductionIt helps film and short video creators add new perspectives and enhance the appeal of their content.
- Intelligent video conferencingDynamically adjust the meeting perspective to focus on specific areas or people and enhance interactivity.
- Autonomous driving and roboticsGenerate multi-view driving or navigation scenarios for use in training and testing algorithms.
- Education and TrainingCreate multi-perspective teaching videos to help students better understand and learn.