DreamVideo-2 - A zero-sample video customization generation framework jointly launched by Fudan University, Alibaba, and multiple other institutions.

What is DreamVideo-2?

DreamVideo-2 is an innovative zero-shot video customization framework jointly developed by Fudan University, Alibaba Group, and other institutions. DreamVideo-2 can generate videos with specific themes and precise motion trajectories from a single image and a sequence of bounding boxes, without requiring fine-tuning during testing. The framework learns the appearance of the theme using a reference attention mechanism and controls the motion trajectory based on a binary mask derived from the bounding boxes, achieving precise motion control. DreamVideo-2 introduces hybrid mask reference attention and reweighted diffusion loss to enhance theme representation and balance theme learning with motion control. The research team supported zero-shot video customization tasks using the WebVid-10M dataset and internal data. DreamVideo-2 outperforms existing state-of-the-art methods in both theme customization and motion control, demonstrating its potential and practicality in the field of video customization.

Main functions of DreamVideo-2

• Zero-sample customizationIt requires no fine-tuning during testing and can directly generate videos with specific themes and precise motion trajectories based on a single image and bounding box sequence provided by the user.
• Thematic LearningBased on the reference attention mechanism, the model's inherent ability is used to learn and generate the appearance of a specific topic.
• Precision motion control: Use a binary mask derived from the bounding box to control the motion trajectory of the subject in the video, achieving precise motion control.
• Hybrid Mask Reference AttentionA hybrid latent mask modeling scheme is used to enhance the topic representation of target locations and improve the accuracy of topics.
• Reweighted diffusion lossDistinguish the contributions of the areas inside and outside the delineation frame to ensure a balance between thematic learning and motor control.

The technical principles of DreamVideo-2

• Reference attention mechanism:
  • Based on the multi-scale feature extraction capability of the video diffusion model, a single-topic image provided by the user is used as a single-frame video input model to obtain topic attention features.
  • By incorporating topic features into video features using residual cross-attention, the model's ability to learn specific topics is enhanced.
• Mask-guided motion module:
  • Convert the user-defined bounding box sequence into a binary mask sequence to indicate the motion trajectory of the subject in the video.
  • The design incorporates a mask-guided motion module that includes a spatiotemporal encoder and a spatial ControlNet, extracting motion information from a mask sequence to achieve precise control.
• Hybrid Mask Reference AttentionIntroducing hybrid latent mask modeling into reference attention enhances the subject representation of the target location by adjusting the weights of the background in the mask.
• Reweighted diffusion loss: Design a new loss function that balances topic learning and motion control by adjusting the loss contribution weights of regions inside and outside the bounding box.
• Training and Reasoning:
  • During the training phase, the original 3D UNet parameters are frozen, and newly added components, such as hybrid mask reference attention and mask-guided motion modules, are trained together.
  • During the inference phase, users provide a theme image and a sequence of bounding boxes, which can generate a customized video without additional fine-tuning or attention map modifications.
• Dataset Construction: Construct a new single-topic video dataset containing video masks and bounding boxes, generate annotations using Grounding DINO, SAM, and DEVA models, and support zero-shot video customization tasks.

DreamVideo-2 project address

• Project official website:dreamvideo2.github.io
• arXiv technical paper:https://arxiv.org/pdf/2410.13830

Application scenarios of DreamVideo-2

• Entertainment and social mediaUsers can quickly generate video content containing specific people or objects based on their personal preferences for sharing on social media or for personal entertainment.
• Film and Video ProductionFilmmakers use DreamVideo-2 to preview special effects or scenes, or generate specific action sequences, reducing the cost and time of actual shooting.
• Advertising and MarketingBased on customized video content, businesses can create more engaging ads, using specific products or brands as themes to enhance the personalization and interactivity of their ads.
• Education and TrainingEducational institutions use DreamVideo-2 to generate instructional videos that simulate specific experimental processes or historical events, enhancing the learning experience.
• News and ReportsNews organizations can quickly generate visual content for news stories, especially for events that are difficult to film on-site.